As an example of good prompt-engineering for modeling:
- Avoid Data Leakage
- Avoid Central Tendency Bias (Regression to the mean)
- Use strict in sample and out of sample methods
- Avoid overfitting?
- Incorporate backtesting?
Avoiding Central Tendency Bias:
- Score-range specific features
- Non-centered parameterization
- Changed how team offense and defense parameters were modeled in the hierarchical structure
- Wider prior distributions
One thing that I’m trying to think through a bit more is how to keep things on track. I’ll have a PRD in place, and then things will go off the rails a bit. Maybe a cursor rule?
Turning this into a rule to help with agentic-systems
Pinecoder has a nice bit on the phase this page comes from: pasting code into a Jupyter notebook in a domain they knew nothing about, “simultaneously learning the APIs and the domain while building something.” Hallucinations showed up “at the boundary” of what the model knew, which made them a signal for where to dig in yourself. The “keep things on track” worry above has mostly turned into a question of which checks to build. See more-ambitious-software.