19 Oct 2024
I just built my first Neural Network: Here's my framework for learning in public
I recently joined a research team building time series Transformer models and have become infatuated with the field of deep learning.
As a former trader, turned data engineer, I am now trying to understand the AI side of things.
And this week I just hit my first significant milestone: building my first neural network from scratch, using no machine learning libraries.
Today, I want to share this milestone and offer you my framework for how I decided to learn deep learning in public.
(Here’s my GitHub repo (https://github.com/bitsofchris/deep-learning) and the XOR neural network (https://github.com/bitsofchris/deep-learning/blob/main/code/000_neural-network-xor/nn_SGD.py)).
The Key: Invest in the basics
Knowledge compounds over time.
When you understand the basics well, you gain the freedom and flexibility to explore more advanced concepts creatively.
You have a strong foundation to build upon.
Taking the time to stop your task and go look up something you don’t quite know, especially if it’s something foundational that you will see again, is an investment in your future self.
This is the key concept to understand to unlock the value of life long learning.
When you see the compounding effect of knowledge - you look for opportunities to know something well, to learn it deeply.
Slow down, and focus on the fundamentals.
Why I love learning in public
I've chosen to share my notes and code for this learning project on GitHub (https://github.com/bitsofchris/deep-learning).
This "learning in public" approach is better than learning on your own, but it requires a little more time in sharing what you do. It offers several benefits:
1. Accountability: Sharing your work creates a forcing function, encouraging you to go the extra mile in understanding and polishing your knowledge.
2. Continuous improvement: When you know you'll be sharing your learnings regularly, you start to notice learning opportunities in your daily life.
3. Networking: By putting your work out there, you connect with like-minded individuals, potential mentors, and future colleagues. My previous writing actually played a role in landing me on my current AI research team.
4. Knowledge retention: Externalizing your notes, whether in a private second brain (https://bitsofchris.com/p/start-your-second-brain-a-quick-guide) or a public GitHub repo, helps solidify your understanding and creates a valuable resource that gets exponentially more valuable as you use it.
My framework for learning in public
Inspired by Scott Young's book "Ultralearning (https://www.scotthyoung.com/blog/ultralearning/)," here’s my framework for difficult learning projects:
1. Set a big, exciting goal
Start with a project that genuinely excites you.
For me, it's building deep neural networks for financial data, leveraging my background in day trading.
Your goal should be challenging enough to push you out of your comfort zone but aligned with your interests and expertise.
2. Break it down into milestones
Divide your big goal into smaller, manageable milestones.
My first milestone was implementing a basic neural network from scratch to solve the XOR problem.
Having these intermediate goals helps maintain motivation and provides a sense of progress.
3. Focus on a few high-quality sources
Avoid information overload (and the stress that comes with it).
Choose 1-3 reliable resources and stick with them. Even when things get difficult.
Ignore everything else.
4. Balance theory with practice
Adopt a "just-in-time" learning approach instead of drowning in prerequisites.
Start with what excites you most, and fill in knowledge gaps as you encounter them. This approach maintains motivation while ensuring you still build a solid understanding as you go.
When you're not actively coding or building, practice active recall by explaining concepts in your own words.
This technique, inspired by the Feynman method, helps identify areas where your understanding is lacking.
But it also provides a sense of action when you are studying theory.
5. Be consistent
Practice daily, even if it's just for 5-30 minutes.
I aim for six days a week, taking Sundays off.
Promise yourself at least 5 minutes, this will get you past that initial wall of getting started.
My first neural network: A brief reflection
Implementing a neural network from scratch to solve the XOR problem was immensely satisfying.
While the network itself is simple, the process of building it deepened my understanding of the core concepts behind neural networks.
The journey wasn't always linear – I often found myself circling back to revisit concepts I didn't fully grasp at first. But this persistence paid off, and looking back, it's amazing to see how much I've learned in just a few weeks.
Again if you are interested in the actual path I took, follow my deep learning work on GitHub (https://github.com/bitsofchris/deep-learning).
Start your own learning in public project
If there’s something you want to pursue, give this framework for learning in public a try.
* Start by identifying your exciting project and break it down into milestones.
* Find 1-3 resources, and focus on these.
* Commit to 5 minutes daily practice - balancing learning with doing.
Remember, knowledge compounds over time. The key is just to consistently build on what you have.
Thanks for reading and happy learning!
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit bitsofchris.com (https://bitsofchris.com?utm_medium=podcast&utm_campaign=CTA_1)
