Flexible styling of declarations

[jakobandersen]

Currently there are very few options for styling declarations, essentially only a few desc_* nodes in sphinx/addnodes.py. For some languages (e.g., C++) declarations may contain many keywords and built-in types that should be coloured to help readability. It would be cool if themes could define custom styles.

I'm not sure what the best way to go about this is, but how about the following.
From Pygments (pygments.tokens) I found the following types of tokens:
'Comment', 'Error', 'Escape', 'Generic', 'Keyword', 'Literal', 'Name', 'Number', 'Operator', 'Other', 'Punctuation', 'STANDARD_TYPES', 'String', 'Text', 'Token', 'Whitespace'
To addnodes.py we could add additional desc_* nodes for those we don't have yet, and that makes sense. Each should be constructible with a domain name. In terms of CSS each of them should emit two classes; a generic one and a domain-specific one (if a domain is specified).

• Can this be done in a format-agnostic way with Docutils? (via the classes attribute?)
• Can we make the nodes inherit from a common class so writers can opt-in to support this?
• As this overlaps a lot with Pygments, is there anything we can reuse? E.g., the style maps?
• Is there a better way to implement this?
• Are people against this feature?

[jakobandersen]

In https://github.com/jakobandersen/sphinx/tree/4289_decl_styling there is a proof-of-concept implementation, based on the ideas outlined above (The current changes simply colours the C++ keyword template blue in the HTML output):

• There is a single new node type for text elements in declarations (to avoid a lot of boiler-plate code in each writer).
• The node type is not supposed to be constructed directly, but by functions for each syntactic construct.
• The node always adds 2 or 3 elements to the classes attribute of it self (are the class names appropriate?):
  • d, a generic class for all description elements.
  • A class specific for the syntactic construct (e.g., k for keywords).
  • If a domain is specified, also <e>-<domain> where <e> is the is the element above, and <domain> is the given domain (e.g., k-cpp for a keyword in the C++ domain).
• Only the HTML formats are actually affected by these changes, though I expect a set of macros for the Latex format could be introduced, controlled by checking the classes attribute in the Latex writer.
  Maybe @jfbu already have some thoughts of how to do this best?
• The nodes desc_name and desc_addname can be deprecated and be made as their own syntactic constructs.

Modifying a domain to use this new node type is a lot of work, so before making a larger pull-request: are people ok with this change?

[jfbu]

Maybe @jfbu already have some thoughts of how to do this best?

@jakobandersen can you check https://github.com/jfbu/sphinx/tree/4289_decl_styling?

It is not "how to do it best" but rather "how to do it fast" ;-)

(I have amended the commit message)

edit: in my commit I do a manual definition of LaTeX styling macro. But from brief look at Pygments, the Formatters have get_style_defs() methods which produce automatically the styling (css for html, latex commands for latex), starting from a Style.styles. I have not checked if Pygments offer a user interface to extend the list of STANDARD_TYPES (in pygments.token), but this can surely be done. Thus producing CSS style and LatTeX macros can be obtained from Pygments itself. Currently Sphinx LaTeX Builder uses Pygments on the fly to produce again on each build the highlighting macros. It should not be too involved to get this to include the added highlighting macros for cpp domain.

[jfbu]

The commit message at jfbu@441c3de makes a mention of + character to be avoided but anyhow it is not legit (without escaping) in CSS class names or selectors, so this is (mainly) not an issue.

(I have again amended the commit message)

edit: the commit message at jfbu@cc632ba discussed some issue of usage of \sphinxcode LaTeX macro. I have pushed jfbu@88cf9be to address that.

[jakobandersen]

@jfbu, thanks, looks quite nice. I will take a more thorough look when I find more time.
A few thoughts:

• It sounds like Pygments already have all the functionality for format-agnostic style specification. I vote for directly using that as much as possible.
• Though, we should be able to use the highlighting directly, without the lexer.
• If we anyway use Pygments, then we can simply have a normal attribute on the Docutils node, instead of using the classes attribute.
• We have a few more types than Pygments due to the desc_name and desc_addname types,
  probably we can add some default values when a traditional Pygments style is selected.
• There should probably be a global style config variable, but the possibility to overwrite for specific domains.
• Each of those config variables should work very similarly to the pygments_style variable.
• This means that multiple Pygments styles needs to be included in the same document, and used in different contexts. This seems to be possible via the get_style_defs method (at least for HTML and Latex).
• There should be builtin styles that emulate the current look.
• I guess custom styles by users can be defined rather easily through extensions already?

[jfbu]

(briefly to say that for the LaTeX aspects, I have merged current master in my forked 4289_decl_styling branch because master now includes #4370 which was motivated by needs of this discussion)