basic example

dist/index.html

@@ -294,7 +294,7 @@ if self.config._attn_implementation != "eager":
 </code></pre>
 <p>We often read and understand that <code>kwargs</code> are criticized, and we are typing them however we can, but we cannot enforce them all the time because other libraries such as vLLM don’'t use the same kwargs.</p>
 <p>It is a strength of the new attention interface, where it can be plugged in various backends, because most of the signature is not enforced. We INFORM but do not ENFORCE. That way, the current system is a <a href="#minimal-user-api">minimal user api</a>.</p>
-<p>For a better <em>information</em>, we plan to use <code>python</code>features such as <code>Annotated</code> for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the type annotations, telling for instance the expected dimensions and contents of a tensor.</p>
+<p>For better <em>information</em>, we plan to use <code>python</code> features such as <code>Annotated</code> for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the type annotations.</p>
 <h2><a id="simpler-tensor-parallelism"></a> Simpler Tensor Parallelism</h2>
 <p>We want to touch minimally to the modeling code, and only modify it when <em>architectural changes</em> are involved. For instance, for tensor parallelism, we instead now specify a simple <code>tp_plan</code>.</p>
 <h2><a id="layers-attentions-caches"></a> Layers, attentions and caches</h2>
@@ -305,7 +305,7 @@ if self.config._attn_implementation != &quot;eager&quot;:
 class GlmRMSNorm(nn.Module):
     ...
 </code></pre>
-<p>Plus, this opened another angle of contribution for the community. People who are GPU whisperers can check on the <a href="https://huggingface.co/blog/hello-hf-kernels">kernel community blog post</a> to learn more about it!</p>
+<p>Plus, this opened another angle of contribution for the community. People who are GPU whisperers can now contribute optimized kernels. You can check on the <a href="https://huggingface.co/blog/hello-hf-kernels">kernel community blog post</a> to learn more about it!</p>
 <h2>The good modularity</h2>
 <p>Now, we have a form of inheritance in our codebase. Some models become standards, and model contributors are given the opportunity to <em>define standards</em>. Pushing the boundaries of scientific knowledge can translate into the boundaries of engineering if this effort is made, and we’re striving for it.</p>
 <p>My capacity for abstraction is not that great, compared to other computer scientists and engineers: I need to look at little doodles and drawings, especially when components pile up.</p>
@@ -481,7 +481,19 @@ machinery is the <code>attention mask</code>, cause of confusion. Thankfully, we
 <p><img src="static/model_debugger.png" alt="Model debugger interface"></p>
 <h3>Transformers-serve</h3>
 <p>Having all these models readily available allows to use all of them with transformers-serve, and enable interfacing with them with an Open API-like pattern.</p>
-<p>#### add example</p>
+<pre><code class="language-bash"># Start serving a model with transformers serve
+transformers serve microsoft/DialoGPT-medium --port 8000
+
+# Query the model using OpenAI-compatible API
+curl -X POST http://localhost:8000/v1/chat/completions \
+  -H &quot;Content-Type: application/json&quot; \
+  -d &quot;{
+    \&quot;model\&quot;: \&quot;microsoft/DialoGPT-medium\&quot;,
+    \&quot;messages\&quot;: [{\&quot;role\&quot;: \&quot;user\&quot;, \&quot;content\&quot;: \&quot;Hello, how are you?\&quot;}],
+    \&quot;max_tokens\&quot;: 50
+  }&quot;
+</code></pre>
+<p>This provides an OpenAI-compatible API with features like continuous batching for better GPU utilization.</p>
 <h2>Community reusability</h2>
 <p>Adding a model to transformers means:</p>
 <ul>
@@ -489,7 +501,7 @@ machinery is the <code>attention mask</code>, cause of confusion. Thankfully, we
 <li>usable in vLLM, SGLang, and so on without additional code.</li>
 </ul>
 <p>## Inner cooking: CUDA Warmup</p>
-<p>Having a clean <em>external</em> API allows us to work on the true inner workings of transformers. One of the few recent additions was the <em>CUDA warmup</em> via <code>caching_allocator_warmup</code> which improved massively the loading time by pre-allocating GPU memory to avoid malloc bottlenecks during model loading.</p>
+<p>Having a clean <em>external</em> API allows us to work on the true inner workings of transformers. One of the few recent additions was the <em>CUDA warmup</em> via <code>caching_allocator_warmup</code> which improved massively the loading footprint by pre-allocating GPU memory to avoid malloc bottlenecks during model loading.</p>
 <p><div style="border: 1px solid #e2e8f0; border-radius: 8px; background: white; margin: 1.5rem 0;">
     <div style="padding: 1rem; border-bottom: 1px solid #e2e8f0; background: #f8f9fa;">
         <h4 style="margin: 0 0 0.5rem 0; color: #495057;">🚀 CUDA Warmup Efficiency Benchmark</h4>