Getting Blendshapes: How to Bring Face Tracking Data From VTubeStudio Into Your Own Program
This is my documentation.
00:00 Introduction
00:21 Part I: Network Communication
01:10 Computers are connected
02:11 How VTube Studio Communicates
03:29 Timing
05:30 Part II: This is Graphic Stuff
07:03 cgltf to the rescue
07:28 "Shape Keys" (in Blender)
09:12 (cgltf) Data Structure (Attributes/Hierarchy)
10:24 Side note about Buffers, Vertex Buffers
11:11 The Hierarchy Comparison
11:32 Models in Raylib
12:35 Part III: Blendshapes (finally)
14:37 glTF Extensions
15:38 How Blendshapes are Computed
16:09 The Loop
17:12 Most applications, ever.
18:49 Optimizing: The Beginning
19:31 Part IV: Multithreading
21:10 Closing Remarks & Bug Fixes
22:02 I'm Done!
22:22 Future Directions
Send Message to Port 21412: {\"messageType\":\"iOSTrackingDataRequest\",\"sentBy\":\"SoftwareName\",\"sendForSeconds\":1,\"ports\":[50506]}
OpenMP (in Visual Studio): https://learn.microsoft.com/en-us/cpp/build/reference/openmp-enable-openmp-2-0-support?view=msvc-170
VSeeFace: https://www.vseeface.icu/
VTubeStudio Wiki: https://github.com/DenchiSoft/VTubeStudio/wiki
Sending Data to VSeeFace: https://github.com/DenchiSoft/VTubeStudio/wiki/Sending-data-to-VSeeFace
glTF Morph Target Spec: https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_017_SimpleMorphTarget.md
glTF No Morph Targets, use Extensions: https://github.com/KhronosGroup/glTF/issues/1036
Raylib: https://github.com/raysan5/raylib/tree/master
RayGUI: https://github.com/raysan5/raygui/blob/master/src/raygui.h#L2935
Assigned TCP/UDP Ports: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
cGLTF: https://github.com/jkuhlmann/cgltf/blob/master/cgltf.h
JSMN C JSON parser: https://github.com/zserge/jsmn
time.h: https://cplusplus.com/reference/ctime/
Visual Studio Debugger is sometimes wrong, apparently: https://developercommunity.visualstudio.com/t/debugger-shows-data-with-incorrect-struct-members/1225526
2
views
The 5,1 Mac Pro - Could You and Should You run one
This is the kind of computer you explicitly know you want; you don't head out to buy a computer and end up with a cMP.
My voice is still thrashed from my friends Holloween party.
...
I'm starting to think Apple is using "inflated" RAM/Storage upgrade costs to subsidize the M-series chips in the entry level machines.
The NRE on those things has to be super high, and with non-volatile storage these days being so fast it doesn't impact the user experience...
New graphics APIs also let the GPUs directly pull data from your storage device through the PCIe bus.
00:00 Why would I buy this?
01:39 Can you? Yes*
02:54 Choose: Works Well or Is Flexible
03:10 It's old & outdated...
04:15 Got it working
05:02 Should you? No*
06:38 A Unit of Engineering Effort
06:58 Security & Ageing
07:58 If you *really* want one...
Greg Gant's Definitive Upgrade Guide: https://blog.greggant.com/posts/2018/05/07/definitive-mac-pro-upgrade-guide.html
OpenCore Legacy Patcher Walkthrough: https://dortania.github.io/OpenCore-Legacy-Patcher/HOW.html#boot-process-with-opencore
Manual Guide: https://forums.macrumors.com/threads/manually-configured-opencore-on-the-mac-pro.2207814/
cMP: https://everymac.com/systems/apple/mac_pro/specs/mac-pro-twelve-core-2.66-mid-2010-westmere-specs.html
https://en.wikipedia.org/wiki/Mac_operating_systems
SSD: https://www.bestbuy.com/site/samsung-990-pro-2tb-internal-ssd-pcle-gen-4x4-nvme/6523595.p?skuId=6523595&extStoreId=1031&ref=212&loc=1&gclid=Cj0KCQjwtJKqBhCaARIsAN_yS_lPfMOVIAU5pyCc7Ckloi5FMp2POyMDqbZRTq6wUGyPHS-zwXdt9VkaApbjEALw_wcB&gclsrc=aw.ds
92
views
Notes on Computers From First Principles- 2 - The Boot Process
I get the sense this could be better. You can't really...*feel* the design decisions.
Maybe I'll have to redo it sometime.
00:00 I thrashed my voice...
00:15 THE RUNTIME ENVIRONMENTTT
00:59 Going From Ideal to Real
02:17 Model Expansion
03:00 How does this thing boot.
03:56 Splitting up the process
06:32 Technology Progressed
08:01 An Example of Said Progress
09:25 An Unqualified Note on Security
Old Punchcard Bootloading: https://www.righto.com/2021/02/an-ibm-1401-mainframe-computer-at.html
Book on BIOS to UEFI: http://ftp.kolibrios.org/users/seppe/UEFI/Beyond_BIOS_Second_Edition_Digital_Edition_(15-12-10)%20.pdf
Apple Security Document: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf
https://en.wikipedia.org/wiki/Booting
https://en.wikipedia.org/wiki/Programmable_ROM
https://en.wikipedia.org/wiki/EPROM
https://en.wikipedia.org/wiki/Flash_memory
https://en.wikipedia.org/wiki/Vault_7
5
views
Second attempt at simulating waves--with obstacles
I fear I com across as overly didactic. I swear, I can change!
...though honestly, I doubt I'll ever be able to channel the sheer chaos of Evan Kale or Michael Reeves. Or any other v-tuber.
Here are some different waves of simulating how waves travel. I realize that the behavior of the wavefront at 2:53 and 6:53 is incorrect; I fixed it before generating the final output.
These methods actually fill every pixel.
0:00 Beginning
1:34 The issue with raycasting
3:48 Ray back-calculation (Newtons Method .gif)
5:40 Output 1 and the 2nd Method
7:53 Mildly borked graphics
2
views
I designed my own (very basic) computer
Man my skills have gotten rusty.
don't tell the boss
0:00 Intro and Architecture
2:58 CPU: How it works
6:02 What is an FPGA
10:12 Writing a program for my processor
13:33 Debugging
1
view
The turret video (I claim) everyone makes
nifty
I think I'll go play some video games. I picked up a PlayStation bundled with a couple God of War games recently.
00:00 Start
0:20 DC Motors
0:52 On the scale of easy to "black magic"
1.30 Stepper Motors
2:00 Tinkering
2:32 The Turret!
3:00 Arduino IDE and coding stuff
3:55 Fox was given a battery controller! Fox is free (from the PSU)
4:23 Lithium
5:04 Concluding Remarks
5:29 Reflecting and regret
I think I had too much of that eggnog.
5
views
I discover Raylib and started simulating waves with it
Raylib has quickly become my favorite graphics library. Some of my long-term projects require an understanding of waves and some wave
simulations, so...here we are.
The Raylib homepage is here: https://www.raylib.com/
And cheatsheet: https://www.raylib.com/cheatsheet/cheatsheet.html
I'm looking forward to playing around with it some more.
If you do want to learn openGL, I really like this tutorial: https://learnopengl.com/
1
view
Reading a DICOM image to generate a 3D Model
This was my initial attempt--if which I accomplished half of what I set out to do. There's a lot I want to do to both improve the model with more source data, and to increase the degree to which I can interact with it.
DICOM Standard: https://www.dicomstandard.org/current
Roni's blog: https://dicomiseasy.blogspot.com/
Computerphile on Blurring (Dr. Pound is great): https://www.youtube.com/watch?v=C_zFhWdM4ic
Computerphile on Sobel Filter: https://www.youtube.com/watch?v=uihBwtPIBxM
8
views
First shot at programming with the Kinect (and Raylib)
I have this sneaking feeling I'm gonna be doing all the trigonometry by hand at some point. Also, tried upgrading my model and just ended up reintroducing some mesh clipping.
Fun stuff.
The Kinect Tutorial: https://ed.ilogues.com/Tutorials/kinect2/kinect0.html
Raylib: https://www.raylib.com/index.html
Setting up Raylib in Visual Studio: https://github.com/raysan5/raylib/wiki/Create-Visual-Studio-2019-Project
Kinect for Windows SDK 2.0 https://docs.microsoft.com/en-us/previous-versions/windows/kinect/dn758354(v=ieb.10)
OpenGL Types: https://www.khronos.org/opengl/wiki/OpenGL_Type
C Types: https://en.wikipedia.org/wiki/C_data_types
5
views
I'd like to learn guitar someday, but got distracted writing this little teaching aid
Honestly not sure I even have time to practice another instrument.
The video I'm referencing is here: https://www.youtube.com/watch?v=PJddQ6Q0UDo
4
views
A Summary of the Five Worlds [Mythos 1]
There are only 5. Sorry for referencing some technical stuff for an introduction.
Also, forgot to address one potential question, so I'll put it here: what happens when I summon a variation of someone/something from another timeline? What happens when I *go* to another timeline?
- For Multiverse travel: Long story short, nothing is traveling between worlds; instead you are manifesting one possibility from the possibility space into our world by rejiggering the *Psi* in a region of space-time-orthogonality. In the case of you going someplace, you likely are creating an unstable pocket dimension ("between" the 5 stable points in the configuration space) and are shifting yourself into it. Make sure you get back before it collapses.
- If you introduce a temporal shift into the manifested pocket dimension (and most attempts at time travel are not actual time travel, but rather generating a copy of our world from a different point in history), nothing happens. But if you introduce temporal dilation you have to account for that in some way--a rudimentary model would be the Temporal Flux model, where you represent a discontinuity in time at some boundary as generating "flux". A first order approximation would be the Steiner Flux Equilibriation Theory, which posits that any discontinuous temporal flux at a boundary will take the path of least time [...] to equilibriate. Basically, temporal boundaries are inherently unstable and will collapse naturally and catastrophically.
Actually sending yourself back in time can't be done safely. Do not attempt to do so.
5
views
Overview of How Computers Work
A brief description of what computers do and how they do it. I think it's a good framework to have if you're writing code, even if it's not essential to software development.
I'll probably remake this video when I improve my model, animations, and just...everything?
A Brief Introduction to nCurses: Making a Menu - 1
An introduction to the nCurses C++ library, which can be used to actively control the terminal input and output. Here I write a basic menu. There may be a part 2.
There are some interesting concepts that appear here that are also used in game engines; namely, the frame buffer and its interplay with the actual screen.
Files: https://github.com/FranciscoJintoFox/nCursesExamples
[Rambling] Note son ML: Conceptual Introduction
I stumbled through this process on my own a few years ago, trying to figure out something for a personal project. Turns out these geniuses figured it out half a century ago and I'm just late to the party (눈_눈)
00:00 Startup
00:24 The MACHINE
01:26 Definition and Economics Tangent
01:44 The Computer IS a machine
02:19 ALGORITHMS
03:08 The Inventions of Man tend to expand in applicaiton...
03:38 Laying the foundation
04:40 Payoff
05:39 What the Pros (Mitchell) Say(s)
07:11 Concluding remarks
References:
Machine Learning by Mitchell: https://www.cs.cmu.edu/~tom/mlbook.html
Deep Learning text: https://www.deeplearningbook.org/
Edward Thorp Card Counting Paper: http://www.edwardothorp.com/wp-content/uploads/2016/11/AFavorableStrategyForTwenty-one.pdf
Economics book is Rothbard's Man, Economy, and State (though it's not my first recommendation):https://mises.org/library/man-economy-and-state-power-and-market
Cancer Cell ML Detection Research:
https://www.cancer.gov/news-events/cancer-currents-blog/2022/artificial-intelligence-cancer-imaging
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9504738/
https://www.nature.com/articles/s43856-022-00199-0
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8321208/
20
views
Working/Coding with glTF
You may think the crumpling is me leaving the frame, but that's just how it is sometimes.
¯\_(ツ)_/¯
0:00 Corny Intro
0:33 glTF and VRM Specifications
2:12 glTF is a last mile format
4:11 Abstractions!
5:20 Skinning
6:33 Memory Stuff
7:15 I want to control the model.
8:05 Notes for future me
9:50 Failures during debugging.
glTF Reference Guide:
https://www.khronos.org/files/gltf20-reference-guide.pdf
glTF Specx/Info:
https://reistry.khronos.org/glTF/specs/2.0/glTF-2.0.html#skins
https://github.com/KhronosGroup/glTF
https://docs.fileformat.com/3d/gltf/
https://www.khronos.org/gltf/
glTF Tutorials
https://github.com/KhronosGroup/glTF-Tutorials
https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_002_BasicGltfStructure.md
glTF Structure
https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/images/gltfJsonStructure.png
glTF Skins Tutorials
https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_019_SimpleSkin.md
https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_020_Skins.md
Random Stack Overflow Answers/Spec Hints:
https://webglfundamentals.org/webgl/lessons/webgl-skinning.html
https://stackoverflow.com/questions/64904889/what-is-the-correct-mapping-of-inverse-bind-matrices
https://stackoverflow.com/questions/64690136/can-you-get-an-explicit-skeleton6-out-of-a-gltf-file
https://math.stackexchange.com/questions/360286/what-does-multiplication-of-two-quaternions-give
https://github.com/KhronosGroup/glTF/issues/1674
https://ogldev.org/www/tutorial38/tutorial38.html
https://stackoverflow.com/questions/55989429/understanding-the-skinning-part-of-a-gltf2-0-file-for-opengl-engine
Finding a Quaternion Rotation
https://stackoverflow.com/questions/1171849/finding-quaternion-representing-the-rotation-from-one-vector-to-another
https://personal.utdallas.edu/~sxb027100/dock/quaternion.html
cgltf
https://github.com/jkuhlmann/cgltf
tinygltf:
https://github.com/syoyo/tinygltf/issues/159
JSON File Format
https://docs.fileformat.com/web/json/
memcpy man:
https://cplusplus.com/reference/cstring/memcpy/
Raylib Setup:
https://github-wiki-see.page/m/raysan5/raylib/wiki/Create-Visual-Studio-2019-Project
https://github.com/raysan5/raylib
Kinect Tutorial:
https://ed.ilogues.com/Tutorials/kinect/kinect0.html
A Potentially Useful Paper:
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/BodyPartRecognition.pdf
https://gamerant.com/video-games-90s-best/#gran-turismo-ndash-1997
14
views
J100 - You have to regularly change your vehicles oil
So many reflections.
I like ChrisFix' guides:
https://www.youtube.com/watch?v=O1hF25Cowv8
https://www.youtube.com/watch?v=t7JCh7PHoDc
00:00 Introduction
0:43 Why Does Oil Degrade?
01:36 A Vague, Yet Excessively Detailed Chemistry Tangent
02:45 Additives
03:24 The Actual Process
Some References
https://youtu.be/RcEDc9HraIU
https://www.machinerylubrication.com/synthetic-oil-31800
https://en.wikipedia.org/wiki/Motor_oil
https://en.wikipedia.org/wiki/Base_oil
https://en.wikipedia.org/wiki/Synthetic_oil
https://www.machinerylubrication.com/Read/29169/oil-degradation-cause
http://butane.chem.uiuc.edu/pshapley/GenChem2/A10/2.html
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6600360/
https://www.machinerylubrication.com/Read/532/around-around-where-oil-goes-in-your-engine
https://en.wikipedia.org/wiki/Oil_additive
3
views
My Notes on Linux from First Principles - 1
Some notes on how the desktop environment is actually put together.
00:00 Start
00:22 On Computers
01:50 THE KERNEL *guitar riff*
02:35 Drivers and Things
03:06 What the Kernel Does
03:43 But We Want An Interface
04:07 Leaving the Embedded World
04:57 Console to Desktop
06:51 Creating Your Own Desktop Environment
10:05 Closing Remarks
References:https://lwn.net/Kernel/LDD3/
https://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures/os-dev.pdf
https://www.makeuseof.com/how-to-build-a-linux-desktop-environment/https://www.gnu.org/home.en.html
https://en.wikipedia.org/wiki/Quantum_circuithttps://en.wikipedia.org/wiki/Quantum_computinghttps://en.wikipedia.org/wiki/Generative_pre-trained_transformer
2
views
Camera Turret 2 - Communication Inversion
Beej's Guide to Network Programming: https://beej.us/guide/bgnet/
The C Book: https://www.amazon.com/Programming-Language-Brian-W-Kernighan/dp/0131101633
(You could google it and find a PDF)
1
view
J100 Land Cruiser Starter Replacement
I really hope some kid is watching his favorite idol vtuber when his dad walks past and casually drops "Hey, those vtuber guys are pretty nifty, huh? I get it now."
0:00 Engines are neat
1:34 Start disassembly
2:17 Connectors
3:00 Hoses
5:32 Removing the Intake Manifold Assembly
6:35 I bought a lift
7:19 Remove the starter
7:45 Looking at the starter
9:10 Install the starter
10:00 Cleaning and new gaskets
10:48 Reinstalling the Intake Manifold Assembly
12:22 Reinstall Hoses
12:38 Reinstall Connectors
14:10 Finishing Touches
14:56 The fruits of my labor
15:08 The rest of the story
15:32 It works!
3
views