AI embeddings without OpenAI, $0; tokenization @ Hugging Face API & libs

Hugging Face - Wikipedia

Hugging Face, Inc. is a French-American company that develops computation tools for building applications using machine learning. It is most notable for its transformers library built for natural language processing applications and its platform that allows users to share machine learning models and datasets and showcase their work.

$0 Embeddings (OpenAI vs. free & open source) - YouTube
tutorial by Rabbit Hole Syndrome

Source code: //github.com/rabbit-hole-syndrome/open-source-embeddings

Sentence Embeddings (SBERT): //sbert.net/ MTEB Leaderboard: 

MTEB: Massive Text Embedding Benchmark

MTEB Leaderboard - a Hugging Face Space by mteb

mteb/docs/adding_a_model.md at main · embeddings-benchmark/mteb @GitHub

Byte-Pair Encoding tokenization - Hugging Face NLP Course

WordPiece tokenization - Hugging Face NLP Course

Hugging Face JS libraries (calling API)

Brilliant | Learn interactively (embeddings)

Generate LLM Embeddings On Your Local Machine - YouTube

Let's build GPT: from scratch, in code, spelled out. - YouTube