Free

Your name will be put in backers.md in the GRIN repository.

Backer

Your name will be put at the top of backers.md in the GRIN repository.

Generous Backer

Your name or company logo (small) will be put in backers.md in the GRIN repository.

Bronze Sponsor

Your name or company logo (medium) will be put at
<ul>
<li>the top of backers.md in the GRIN repository</li>
<li>on a dedicated sponsors page on grin-compiler.github.io</li>
</ul>

Silver Sponsor

Your name or company logo (big) will be put at 
<ul>
<li>The homepage of grin-compiler.github.io</li>
<li>The top of the project repo's README</li>
</ul>

Gold Sponsor

General Support

Patron shout-out

Hi,This time I'd like to tell the story why and how I learned to build web UI based compiler tools. More specifically I'll talk about the <a href="https://github.com/grin-compiler/haskell-code-spot" rel="nofollow noopener" target="_blank">haskell code spot</a> project which is a runtime profile data visualizer for Haskell.The project idea dates back to 2017 when I searched for javascript charting libraries to visualize the performance of a python service at work in a nice and interactive way. I was surprised about the abundance and the visual coolness of web UI technology. I realised that Haskell is so much behind the industry standards regarding the tooling user experience. I also thought that this should be changed, in the end it is up to us (Haskell developers) to choose our technology stack. So I decided that I'd like to learn web technology and frontend development sometime in the future just to do cool UI with good UX.<a href="https://gitlab.haskell.org/ghc/ghc/-/wikis/event-log" rel="nofollow noopener" target="_blank">GHC eventlog</a> was the other technology that I discovered in 2017. I have to admit that I almost never profile my haskell programs which is a bad thing. It is just too complicated to recompile the whole project and GHC's profiling and debug tooling is just ridiculously basic and rough. In fact I believe that it is one of the main blockers of Haskell's industrial adoption when it comes to software maintenance. Eventlog is a subsystem of the GHC run-time system that can emit a binary log of various runtime system events (profiling, thread scheduling, GC work, memory allocation, etc). It is the next generation of the GHC profile output format. It is a backward and forward compatible binary format and it encodes more information than the still dominant plain text table formats that show the memory allocation and cost centre cpu usage. While the plain text based (.hp and .prof) formats are human readable, they contain only the fraction or just a projection of the run-time collected statistics. In short, they are lossy formats and the .prof format was designed with <a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/profiling.html#cost-centres-and-cost-centre-stacks" rel="nofollow noopener" target="_blank">human consumption</a> in mind. In contrast, eventlog is intentionally just a container format that is rich with information intended for use by various third party tools.With all this knowledge about eventlog and webtech I imagined a tool that can visualize the available information intuitively. This goal will be accomplished with an iterative approach to development.The first thing that I wanted to have is to show the profiling data directly on the source code. From the UI technology side this requires libraries to show syntax highlighted haskell source code and charting libraries for data visualization. Luckily web technology already provides an easy to use solution for these requirements. In fact there are too many libraries that I can choose from, and it needs quite a bit of work to check the most popular and robust ones to form an opinion and make a decision about the final technology stack. I did some research by reading many blog posts all titled something like: 'top 10 JS charting libraries' and did the same for source code syntax highlight/editor JS libraries. I really enjoyed the abundance of the reviews and forum comments. It helped me to narrow down the selection of libraries. However, I did not want to settle on a single library without trying all the promising ones. So in the end I used <a href="https://d3js.org/" rel="nofollow noopener" target="_blank">D3</a>, <a href="https://c3js.org/" rel="nofollow noopener" target="_blank">C3</a> and <a href="https://plotly.com/javascript/" rel="nofollow noopener" target="_blank">Plotly.js</a> charting libraries and <a href="https://codemirror.net/" rel="nofollow noopener" target="_blank">CodeMirror</a> to show source code in haskell code spot.D3 is the most popular low level (customizable) charting library that is used in data journalism. There are excellent written and video tutorials and courses on the web for it. I learned D3 from a 13 hours long <a href="https://www.youtube.com/watch?v=_8V5o2UHG0E" rel="nofollow noopener" target="_blank">video course</a>. It took a few days to accomplish but it was fun. C3 and Plotly.js are high level charting libraries with a simple API but under the hood both use D3 as a drawing backend. On one hand they are just slightly customizable but on the other hand they require only a <a href="https://c3js.org/samples/data_name.html" rel="nofollow noopener" target="_blank">few lines of code</a> to get a cool looking interactive chart. So C3 and Plotly.js are excellent for prototyping. For the source code view component I also wanted something lightweight and simple. I checked the <a href="https://ace.c9.io/" rel="nofollow noopener" target="_blank">ACE editor</a> and the <a href="https://microsoft.github.io/monaco-editor/" rel="nofollow noopener" target="_blank">monaco-editor</a> component that VS Code also uses. <a href="https://codemirror.net/doc/manual.html#usage" rel="nofollow noopener" target="_blank">CodeMirror</a> has the simplest API from all of these and I only needed the source view features. The other libraries are designed to be feature rich editors primarily, so they would be too heavyweight for a simple syntax highlighter use-case.There is one more reason I wanted to try web programming, it is the abundance of code examples and learning material. I wanted to know how it feels like when I can find detailed answers with sample code for my custom coding problems. I call it 'copy paste programming'. You know stack overflow, blog posts, gist, etc. It is something I have not experienced for a long time. I wanted to know how it feels to be a javascript programmer and web developer. IMO it also helps me understand how a mainstream programmer sees Haskell and its ecosystem.I intentionally wanted to use the state of the art web dev technologies. In 2017 that was <a href="https://reactjs.org/" rel="nofollow noopener" target="_blank">React</a> and I had some experience with it along with vanilla JavaScript. In 202X I'm not sure about React. I've found <a href="https://svelte.dev/" rel="nofollow noopener" target="_blank">Svelte</a> which is an alternative to React and IMO its syntax is highly superior. Essentially it has the syntax of the native web technologies (JS/HTML/CSS) while React (with JSX) just mimics HTML. This is a big advantage because every code snippet and blog post written about JS/HTML/CSS is still valid as it is for Svelte, so the mainstream copy paste programming approach is still applicable. The minor but still important advantage of Svelte is that it generates much smaller and faster code then React. The Svelte learning curve was easy, I watched 2-3 <a href="https://www.youtube.com/watch?v=AdNJ3fydeao" rel="nofollow noopener" target="_blank">presentation videos</a> and completed the Svelte online <a href="https://svelte.dev/tutorial/basics" rel="nofollow noopener" target="_blank">tutorial</a>. Regarding CSS I had only basic knowledge, luckily I've found an excellent <a href="https://www.youtube.com/kepowob/featured" rel="nofollow noopener" target="_blank">youtube channel</a> teaching mainly CSS and HTML. I managed to learn the most important topics like css <a href="https://www.youtube.com/watch?v=IWFqGsXxJ1E" rel="nofollow noopener" target="_blank">units</a>, <a href="https://www.youtube.com/watch?v=Bcr70LIJcOk" rel="nofollow noopener" target="_blank">selectors</a>, <a href="https://www.youtube.com/watch?v=hwbqquXww-U&amp;list=PL4-IK0AVhVjMSb9c06AjRlTpvxL3otpUd" rel="nofollow noopener" target="_blank">flexbox</a>, <a href="https://www.youtube.com/watch?v=plRcoRqLriw&amp;list=PL4-IK0AVhVjPv5tfS82UF_iQgFp4Bl998" rel="nofollow noopener" target="_blank">grid</a> and some other random stuff from youtube videos. It gave me enough confidence to get started with my project with the following technology stack: Svelte, CSS, HTML, D3, CodeMirror.You might ask why I don't use Elm, PureScript or GHCJS for web programming. The short answer is that 'copy-paste programming' would not work with them. The web is full of useful code samples for everyday webdev problems. This helps to lower the entry level for beginners to get started with their projects. When the sample syntax does not match with the project's language then you have to be an expert to translate the JS/HTML/CSS sample to your typed functional DSL which breaks the development flow.Now let’s see how <a href="https://twitter.com/csaba_hruska/status/1287701943863980032" rel="nofollow noopener" target="_blank">haskell code spot works</a>.<img height="256" src="https://lh4.googleusercontent.com/XicY38vB_a9oGbIfI-gs4n2nvdK0c3QXE2xtP6vWZ5F1KcVXJN3eBOtN9NGS3EniTvps9IOAn9kTMzEQgz_MeNQFgutihkat3WiNzwyZbca2pimd5KZMADvCtqLfAEWqDJ5_i9hP" width="451">Its architecture is simple, as it consists of a server written in Haskell with <a href="https://hackage.haskell.org/package/scotty" rel="nofollow noopener" target="_blank">Scotty</a> and a frontend written in Svelte. The frontend is a thin client that visualizes the data it gets from the server. The server has two kinds of data sources: 1. eventlog files (.eventlog) 2. <a href="https://github.com/grin-compiler/ghc-whole-program-compiler-project" rel="nofollow noopener" target="_blank">GHC-WPC</a> project files (.ghc_stgapp)The eventlog is parsed with the <a href="https://hackage.haskell.org/package/ghc-events" rel="nofollow noopener" target="_blank">ghc-events</a> haskell library, converted to JSON with Aeson and then sent to the frontend when requested. The backend does not change the eventlog message format at all, it does not transform the eventlog messages beside some optional message filtering. The backend is also capable of looking up the source code of the modules of the inspected project. This feature relies on GHC-WPC, because the vanilla GHC pipeline does not save information about the project source code location. GHC-WPC generates an .ghc_stgapp file for the executable that has path location data for all the project dependencies. The frontend can query the source code of a specific module name based on the .ghc_stgapp file.In the first iteration of this project I only wanted to do a simple visualization of a subset of the eventlog messages. The memory usage related messages looked to be the simplest option. Each eventlog message has a <a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#t:Event" rel="nofollow noopener" target="_blank">timestamp value</a> that is relative to the application start time and a message specific payload. The payload of <a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:HeapLive" rel="nofollow noopener" target="_blank">HeapLive</a>, <a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:HeapSize" rel="nofollow noopener" target="_blank">HeapSize</a> and <a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:HeapAllocated" rel="nofollow noopener" target="_blank">HeapAllocated</a> messages is an integer that tells the memory consumption in bytes. It was trivial to visualize the timestamp and bytes in D3, C3 and Plotly.js as a line chart. I literally copy-pasted the code samples from the chart lib documentation. The eventlog also marks when the GC does some work (<a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:GCWork" rel="nofollow noopener" target="_blank">GCWork</a>) and in profiling mode it saves the current cost centre stack. The cost centre stack (<a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:HeapProfSampleCostCentre" rel="nofollow noopener" target="_blank">HeapProfSampleCostCentre</a>, <a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:ProfSampleCostCentre" rel="nofollow noopener" target="_blank">ProfSampleCostCentre</a>) eventlog message is an array of 32 bit cost centre identifiers. The cost centres are defined in separate messages. The cost centre specification (<a href="https://hackage.haskell.org/package/ghc-events-0.13.0/docs/GHC-RTS-Events.html#v:HeapProfCostCentre" rel="nofollow noopener" target="_blank">HeapProfCostCentre</a>) identifies the source module and a source location. Then it is up to the user tool how it looks up the module source code.GHC has recently introduced the <a href="https://www.haskell.org/ghc/blog/20190626-HIEFiles.html" rel="nofollow noopener" target="_blank">.hie files</a> to support advanced editor tooling. The .hie file contains a simplified version of the AST including the original module source code and detailed type information. So the .hie files had just the right thing I needed. Unfortunately the format is not stable across GHC versions and sadly it makes .hie files useless in this case. Instead I extended GHC-WPC to save the module haskell source into the external STG binary. Temporarily this is acceptable, I'll do better in the future by introducing a generic container (.zip) file for each module that can store custom data. Then I could store the module haskell source in that container along with the<a href="https://github.com/grin-compiler/ghc-whole-program-compiler-project/tree/master/external-stg" rel="nofollow noopener" target="_blank"> external stg IR</a>.Unfortunately I encountered a problem with eventlog also, it turned out that the last two GHC releases (8.8, 8.10) generate <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/17948" rel="nofollow noopener" target="_blank">corrupted eventlog files</a> due to a <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/17271" rel="nofollow noopener" target="_blank">race condition bug</a> in the RTS. To me this fact is an indicator that <a href="https://twitter.com/csaba_hruska/status/1284782524577787904" rel="nofollow noopener" target="_blank">eventlog is barely used</a> in the real world.Finally I'd like to mention some related projects. The first one is <a href="https://github.com/mpickering/eventlog2html" rel="nofollow noopener" target="_blank">eventlog2html</a>, which is the fancier web based version of the old-school <a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/profiling.html#profiling-memory-usage" rel="nofollow noopener" target="_blank">hp2ps</a> tool. It generates an html file from an eventlog file that looks much better than a static pdf and it is<a href="https://mpickering.github.io/eventlog2html/examples/ghc.eventlog.html" rel="nofollow noopener" target="_blank"> interactive at some level</a>. <img height="443" src="https://lh5.googleusercontent.com/COtbDt-4PSUEF4bJJ93KAm1BGgxuM9W953HyaZjK3Zy_79FQ3r6NkzmU5CFV-qaQIK76b0L_n4-j9uLeKzPxr-rr6iJLH5kidWMpiiWJjX6FsD819Aalc864CiH9JT30ZLS0CLQ8" width="463">Eventlog2html generates a static html in contrast to haskell code spot that has a client server architecture which gives much bigger freedom for fancy data visualization.The other project is <a href="https://github.com/bgamari/ghc-debug/" rel="nofollow noopener" target="_blank">ghc-debug</a>. It aims to allow remote debugging for GHC compiled programs. Its core idea is to add server functionality to the GHC runtime system and then allow other programs like profilers and debuggers to connect and query the program's memory structure with the finest detail. The <a href="https://github.com/bgamari/ghc-debug/blob/master/call-for-participation.md" rel="nofollow noopener" target="_blank">idea is awesome</a>, but the project is in an early stage of development and it requires a development branch of GHC.It would be extremely useful to have a usable haskell debugger and live program profiler. I'd like to work on these indeed, but following a slightly different approach compared to ghc-debug. I'd prefer to implement an stg interpreter first and build a working debugger and profiler for it in the first development iteration. IMO this is a much easier approach and also allows more in-depth inspection of program properties, like value lifetimes.I favour the interpreter approach for the prototyping phase. My motto is that it is better to have a pattern match error than a segmentation fault.

Csaba Hruska