I've been using a paid subscription to ChatGPT Plus since May 2023 and I'm still happy with the usage. It's one of those tools you have to master to get most of the time spent with it. There are a lot of benefits to use it, of course there are also negative aspects and bugs. Some of them can be mitigated, and that's the purpose of the current post.
Disclaimer / Context of use π§Ύ
- I'm ChatGPT mostly for scripts and each conversation is focused on singular topics, i.e single-file codebases if possible.
- I don't use Open API Keys and I'm not interested to use OpenAI directly. I've been on a paid plan for OpenAI and was charged way too much for my needs. I advise anyone to use ChatGPT Plus which remain to this day worth it and good enough both for personal and professional.
- This article was not written nor reviewed by AI/ChatGPT.
Lessons learned - Do β
- Timebox your interactions with ChatGPT. If it takes longer than you expect to reach a solution, stop and do your homework instead.
- Test test test after each change. ChatGPT will make you lose faith by repeating the same mistakes, by removing sensitive code, by providing incomplete solutions, always verify.
- Before running or committing anything generated by ChatGPT, always compare with the previous situation you had. Git diff the result of changes. If you have any doubt about a specific change, submit the git diff to ChatGPT for review and ask for explanations while also explaining why you have doubt (failing tests, errors at runtime, weird code removal, ...).
- Tell ChatGPT to always decouple code in functions.
- Instruct to use minimalist code, and avoid comments in code, respect your specific style (if you provide examples), instruct to write the code that takes the least amount of lines and space and to avoid long functions.
- Keep one conversation per context / type of problem / flow. Once done with the problem, validate the solution (test!), commit locally, and delete the conversation to keep your Chat history clean.
- Try to limit ChatGPT's attention to one single file or one single function at a time, to improve its speed and accuracy. I've experienced inconsistencies and code regressions when expecting ChatGPT to work with too many files and too many changes at once on too many files, as then ChatGPT.
- If you have lot of work you expect ChatGPT to perform for you, be patient and request one change at a time, then review the outcome, and only if it's validated, move to the next. ChatGPT sucks at multitasking. Keep the rest of your TODO list somewhere else to keep track of what remains to be requested from ChatGPT.
- You will be given better results if your tech stack relies on popular libraries and programming languages so keep that in mind and be prepared to deal with more mistakes if you pick niche programming languages.
- Ask to only output the code that require change, e.g add this at the end of your prompt :
just output the function(s) needing change - If you really want to multitask, open multiple ChatGPT sessions (browser tabs) in parallel, each focused on distinct changes, so you can multitask while ChatGPT stays focused on single tasks π
- One change at a time. If ChatGPT keeps making error, step back, restart from to the latest stable version of your code, and ask to make minimal changes, step by step, while you validate each increment.
- When happy with your changes, ask ChatGPT to improve its performance or clean the code.
- When debugging your code with ChatGPT, provide stack traces, inputs, outputs and even complete your input with the copy of the code that seems buggy (based on the stack trace).
- Be suspicious if ChatGPT changes your code way too much or introduce weird changes to dependencies. Review, check, investigate.
- ChatGPT will likely be interesting to use only for writing very small and boring scripts to batch automate some tasks. But you are responsible for the whole, do not forget that. And test. And understand the code. Make it easy to understand and debug. If you can't explain it, rewrite.
- ChatGPT could be good at documenting tasks or at writing tests.
Lessons learned - Avoid β
- Don't trust ChatGPT outputs. OpenAI is known for dreaming and also tends to complexify solutions to simple problems.
- Don't run nor commit anything generated by ChatGPT that you don't understand and always compare ChatGPT's solution with what you had before.
- Don't ask too many improvements or bug fixes at once or be prepared to deal with many new errors and regressions.
- Don't keep conversations for too long, as all the initial context will likely be forgotten about by ChatGPT and you will suffer, also it will cause a lot of scrolling and augment the size of the web page which will be slower to load and will likely crash your browser tab at some point.
- Don't switch context / files / problems in the same conversation. It's a waste of time and you will suffer later when trying to source specific content or make sense of anything.
- Don't share secrets/passwords with ChatGPT.
- Don't waste your time when the LLM seems confused or unable to solve your problem. Either reshape your problem statement or restart from the latest known stable state, ideally in a new conversation. Be confident in your own abilities. Don't be too dependent on any LLM. I once wasted a whole evening and night trying to get ChatGPT to write the solution for me then trying to use it to fix the problems it caused, I was too lazy to code something by myself from the start.
- Don't expect ChatGPT to be as efficient on big codebases and complex problems as it is on small scripts and simple problems. So use it more often for the latter and keep the fun of solving the big problems yourself.
- Don't expect ChatGPT to understand a single thing you do nor why he generates his code. it's a dumb machine without creativity built in. It has to be treated as such and with caution.
- When trying to help you fixing problems caused by its previous solutions, ChatGPT may enter and endless loop of replacing one solution with another, without understanding the context. Before copy pasting anything from ChatGPT, make your own investigation and check the logs of the apps that do not behave as expected, before touching the code. If you find interesting logs, then provide those to ChatGPT or make your own investigation. Do not depend on ChatGPT for too much grunt and research work. You are better than that.
- Don't expect ChatGPT to run well and fast on huge bloated scripts. That should encourage you to decouple your code into functions and modules and specialized files/modules/components/...
Relevant references
- ChatGPT could be used to address the fact that solving some problems is costly in term of engineering. This XKCD meme below should be likely made obsolete if LLMs are used to provide the programs to automate the tasks. For instance when faced with some painful work, I rather use LLM to write me a script to tackle the task. It will be faster than me on some occasions, which itself help removing the need for manual work for tedious tasks, also help removing the need to prompt LLM in the future for that same task, since the script is already provided.
Extras
- If you lose focus on the ChatGPT session in your browser, it's likely the calculation process will interrupt.
- ChatGPT seems stuck at generating the output ? Refreshing the page might be enough, in other cases ChatGPT might automatically continue the generation or will show a button you can hit to force this action.
- I've commented on this topic and this post at https://lobste.rs/s/7a3qhh/how_s_your_experience_so_far_using_llms_for#c_igssaj
- I recommend trying ollama so you can work offline and without feeding all your sensitive data into OpenAI.
Thank you π
Mentions