Widget Overview

The main purpose of a widget is to display a functional control within an HTML page. A widget has a template to generate its own HTML, and a set of parameters that control how it will be displayed. It can also reference resources - JavaScript or CSS files that support the widget.

When defining Widgets, some parameters with be static - they will stay constant for the whole lifetime of the application. Some parameters are dynamic - they may change for every request. To ensure thread-safety, a separate widget instance is created for every request, and dynamic parameters are only set on an instance. Static parameters are set by subclassing a widget. For example:

# Initialisation
class MyWidget(Widget):
    id = 'myid'

# In a request
my_widget = MyWidget.req()
my_widget.value = 'my value'

To make initialisation more concise, the __new__ method on Widget is overriden, so it creates subclasses, rather than instances. The following code is equivalent to that above:

# Initialisation
MyWidget = Widget(id='myid')

In practice, you will rarely need to explictly create an instance, using req(). If the display or validate methods are called on a Widget class, they automatically create an instance. For example, the following are equivalent:

# Explicit creation
my_widget = MyWidget.req()
my_widget.value = 'my value'

# Implicit creation
MyWidget.display(value='my value')


The parameters are how the user of the widget controls its display and behaviour. Parameters exist primarily for documentation purposes, although they do have some run-time effects. When creating widgets, it’s important to decide on a convenient set of parameters for the user of the widget, and to document these.

A parameter definition looks like this:

import tw2.core as twc
class MyTextField(twc.Widget):
    size = twc.Param('The size of the field', default=30)
    validator = twc.LengthValidator(max=30)
    highlight = twc.Variable('Region to highlight')

In this case, TextField gets all the parameters of its base class, tw2.core.widget and defines a new parameter - size. A widget can also override parameter in its base class, either with another tw2.core.Param instance, or a new default value.

class tw2.core.Param(description=Default, default=Default, request_local=Default, attribute=Default, view_name=Default)[source]

A parameter for a widget.

A string to describe the parameter. When overriding a parameter description, the string can include $$ to insert the previous description.
The default value for the parameter. If no defalt is specified, the parameter is a required parameter. This can also be specified explicitly using tw.Required.
Can the parameter be overriden on a per-request basis? (default: True)
Should the parameter be automatically included as an attribute? (default: False)
The name used for the attribute. This is useful for attributes like class which are reserved names in Python. If this is None, the name is used. (default: None)

The class takes care to record which arguments have been explictly specifed, even if to their default value. If a parameter from a base class is updated in a subclass, arguments that have been explicitly specified will override the base class.

class tw2.core.Variable(description=Default, **kw)[source]

A variable - a parameter that is passed from the widget to the template, but cannot be controlled by the user. These do not appear in the concise documentation for the widget.

class tw2.core.ChildParam(description=Default, default=Default, request_local=Default, attribute=Default, view_name=Default)[source]

A parameter that applies to children of this widget

This is useful for situations such as a layout widget, which adds a label parameter to each of its children. When a Widget subclass is defined with a parent, the widget picks up the defaults for any child parameters from the parent.

class tw2.core.ChildVariable(description=Default, **kw)[source]

A variable that applies to children of this widget

Code Hooks

Subclasses of Widget can override the following methods. It is not recommended to override any other methods, e.g. display, validate, __init__.

classmethod Widget.post_define()[source]

This is a class method, that is called when a subclass of this Widget is created. Process static configuration here. Use it like this:

class MyWidget(LeafWidget):
    def post_define(cls):
        id = getattr(cls,  'id', None)
        if id and not id.startswith('my'):
            raise pm.ParameterError("id must start with 'my'")

post_define should always cope with missing data - the class may be an abstract class. There is no need to call super(), the metaclass will do this automatically.


This is an instance method, that is called just before the Widget is displayed. Process request-local configuration here. For efficiency, widgets should do as little work as possible here. Use it like this:

class MyWidget(Widget):
    def prepare(self):
        super(MyWidget, self).prepare()
        self.value = 'My: ' + str(self.value)

Generate the actual output text for this widget.

By default this renders the widget’s template. Subclasses can override this method for purely programmatic output.

The name of the template engine this widget is being displayed inside.

Use it like this:

class MyWidget(LeafWidget):
    def generate_output(self, displays_on):
        return "<span {0}>{1}</span>".format(self.attrs, self.text)

Mutable Members

If a widget’s prepare() method modifies a mutable member on the widget, it must take care not to modify a class member, as this is not thread safe. In general, the code should call self.safe_modify(member_name), which detects class members and creates a copy on the instance. Users of widgets should be aware that if a mutable is set on an instance, the widget may modify this. The most common case of a mutable member is attrs. While this arrangement is thread-safe and reasonably simple, copying may be bad for performance. In some cases, widgets may deliberately decide not to call safe_modify(), if the implications of this are understood.

Widget Hierarchy

Widgets can be arranged in a hierarchy. This is useful for applications like layouts, where the layout will be a parent widget and fields will be children of this. There are four roles a widget can take in the hierarchy, depending on the base class used:

class tw2.core.Widget(**kw)[source]

Base class for all widgets.

class tw2.core.CompoundWidget(**kw)[source]

A widget that has an arbitrary number of children, this is common for layout components, such as tw2.forms.TableLayout.

class tw2.core.RepeatingWidget(**kw)[source]

A widget that has a single child, which is repeated an arbitrary number of times, such as tw2.forms.GridLayout.

class tw2.core.DisplayOnlyWidget(**kw)[source]

A widget that has a single child. The parent widget is only used for display purposes; it does not affect value propagation or validation. This is used by widgets like tw2.forms.FieldSet.

Value Propagation

An important feature of the hierarchy is value propagation. When the value is set for a compound or repeating widget, this causes the value to be set for the child widgets. In general, a leaf widget takes a scalar type as a value, a compound widget takes a dict or an object, and a repeating widget takes a list.

The hierarchy also affects the generation of compound ids, and validation.


In general, a widget needs to have an identifier. Without an id, it cannot participate in value propagation or validation, and it does not get an HTML id attribute. There are some exceptions to this:

  • Some widgets do not need an id (e.g. Label, Spacer) and provide a default id of None.
  • The child of a RepeatingWidget must not have an id.
  • An id can be specified either on a DisplayOnlyWidget, or it’s child, but not both. The widget that does not have the id specified automatically picks it up from the other.

Compound IDs are formed by joining the widget’s id with those of its ancestors. These are used in two situations:

  • For the HTML id attribute, and also the name attribute for form fields
  • For the URL path a controller widget is registered at

The separator is a colon (:), resulting in compound ids like “form:sub_form:field”. Note this causes issues with CSS and will be changed shortly, and made configurable.

In generel, the id on a DisplayOnlyWidget is not included in the compound id. However, when generating the compound id for a DisplayOnlyWidget, the id is included. In addition id_suffix is appended, to avoid generating duplicate IDs. The id_suffix is not appended for URL paths, to keep the paths short. There is a risk of duplicate IDs, but this is not expected to be a problem in practice.

For children of a RepeatingWidget, the repetition is used instead of the id, for generating the compound HTML id. For the URL path, the element is skipped entirely.

Deep Children

This is a feature that helps have a form layout that doesn’t exactly match the database layout. For example, we might have a sinlge database table, with fields like title, customer, start_date, end_date. We want to display this in a Form that’s broken up into two FieldSets. Without deep children, the FieldSets would have to have ids, and field makes would be dotted, like info.title. The deep children feature lets us set the id to None:

class MyForm(twf.Form):
    class child(twc.CompoundWidget):
        class info(twf.TableFieldSet):
            id = None
            title = twf.TextField()
            customer = twf.TextField()
        class dates(twf.TableFieldSet):
            id = None
            start_date = twf.TextField()
            end_date = twf.TextField()

When a value like {'title': 'my title'} is passed to MyForm, this will propagate correctly.


Every widget can have a template. Toscawidgets has some template-language hooks which currently support Genshi, Mako, Jinja2, Kajiki, and Chameleon.

At one point, ToscaWidgets2 aimed to support any templating engine that supported the buffet interface, (an initiative by the TurboGears project to create a standard interface for template libraries). In practice though, there are more differences between template engines than the buffet interface standardises so this approach has been dropped.

The template parameter takes the form engine_name:template_path. The engine_name is the name that the template engine defines in the python.templating.engines entry point, e.g. genshi, mako, or jinja. The template_path is a string the engine can use to locate the template; usually this is dot-notation that mimics the semantics of Python’s import statement, e.g. myapp.templates.mytemplate. Templates also allow specifications like ./template.html which is beneficial for simple applications.

It is also possible to allow your widget to utilize multiple templates, or to have TW2 support any template language you provide a template for. To do this, simply leave the name of the template engine off of the template parameter, and TW2 will select the appropriate template, based on specifications in the TW2 middleware.

For instance, you might have a form.mak and a form.html template (mako and genshi). TW2 will render the mako template if mako is listed ahead of genshi in the middleware config’s preferred_rendering_engines. See the documentation regarding Enabling ToscaWidgets for more information on how to set up your middleware for desired output.

Non-template Output

Instead of using a template, a widget can also override the generate_output method. This function generates the HTML output for a widget; by default, it renders the widget’s template as described in the previous section, but can be overridden by any function that returns a string of HTML.


Widgets often need to access resources, such as JavaScript or CSS files. A key feature of widgets is the ability to automatically serve such resources, and insert links into appropriate sections of the page, e.g. <HEAD>. There are several parts to this:

  • Widgets can define resources they use, using the resources parameter.
  • When a resource is defined, it is registered with the resource server.
  • When a Widget is displayed, it registers resources in request-local storage.
  • The resource injection middleware detects resources in request-local storage, and rewrites the generated page to include appropriate links.
  • The resource server middleware serves static files used by widgets
  • Widgets can also access resources at display time, e.g. to get links
  • Resources can themselves declare dependency on other resources, e.g. jquery-ui.js depends on jquery.js and must be included on the page subsequently.

Defining Resources

To define a resource, just add a tw2.core.Resource subclass to the widget’s resources parameter. It is also possible to append to resources from within the prepare() method. The following resource types are available:

See Resources

Resources are widgets, but follow a slightly different lifecycle. Resource subclasses are passed into the resources parameter. An instance is created for each request, but this is only done at the time of the parent Widget’s display() method. This gives widgets a chance to add dynamic resources in their prepare() method.

Using Your Own Resources

Resources that are defined by pre-existing tw2 packages can be altered globally. For instance, say that you want to use your own patched version of jquery and you want all tw2 packages that require jquery to use your version, and not the one already packaged up in tw2.jquery. The following code will alter jquery_js in not just the local scope, but also in all other modules that use it (including tw2.jqplugins.ui):

import tw2.jquery = "/path/to/my/patched/jquery.js"

Deploying Resources

If running behind mod_wsgi, tw2 resource provisioning will typically fail. Resources are only served when they are registered with the request-local thread, and resources are only registered when their dependant widget is displayed in a request. An initial page request may make available resource A, but the subsequent request to actually retrieve resource A will not have that resource registered.

To solve this problem (and to introduce a speed-up for production deployment), Toscawidgets2 provides an archive_tw2_resources distutils command:

$ python archive_tw2_resources \
    --distributions=myapplication \

Declarative Instantiation

Instantiating compound widgets can result in less-than-beautiful code. To help alleviate this, widgets can be defined declaratively, and this is the recommended approach. A definition looks like this:

class MovieForm(twf.TableForm):
    id = twf.HiddenField()
    year = twf.TextField()
    desc = twf.TextArea(rows=5)

Any class members that are subclasses of Widget become children. All the children get their id from the name of the member variable. Note: it is important that all children are defined like id = twf.HiddenField() and not id = twf.HiddenField. Otherwise, the order of the children will not be preserved.

It is possible to define children that have the same name as parameters, using this syntax. However, doing so does prevent a widget overriding a parameter, and defining a child with the same name. If you need to do this, you must use a throwaway name for the member variable, and specify the id explicitly, e.g.:

class MovieForm(twf.TableForm):
    resources = [my_resource]
    id = twf.HiddenField()
    noname = twf.TextArea(id='resources')

Nesting and Inheritence

Nested declarative definitions can be used, like this:

class MyForm(twf.Form):
    class child(twf.TableLayout):
        b = twf.TextArea()
        x = twf.Label(text='this is a test')
        c = twf.TextField()

Inheritence is supported - a subclass gets the children from the base class, plus any defined on the subclass. If there’s a name clash, the subclass takes priority. Multiple inheritence resolves name clashes in a similar way. For example:

class MyFields(twc.CompoundWidget):
    b = twf.TextArea()
    x = twf.Label(text='this is a test')
    c = twf.TextField()

class TableFields(MyFields, twf.TableLayout):

class ListFields(MyFields, twf.ListLayout):
    b = twf.TextField()

Proxying children

Without this feature, double nesting of classes is often necessary, e.g.:

class MyForm(twf.Form):
    class child(twf.TableLayout):
        b = twf.TextArea()

Proxying children means that if RepeatingWidget or DisplayOnlyWidget have children set, this is passed to their child. The following is equivalent to the definition above:

class MyForm(twf.Form):
    child = twf.TableLayout()
    b = twf.TextArea()

And this is used by classes like TableForm and TableFieldSet to allow the user more concise widget definitions:

class MyForm(twf.TableForm):
    b = twf.TextArea()

Automatic ID

Sub classes of Page that do not have an id, will have the id automatically set to the name of the class. This can be disabled by setting _no_autoid on the class. This only affects that specific class, not any subclasses.

Widgets as Controllers

Sometimes widgets will want to define controller methods. This is particularly useful for Ajax widgets. Any widget can have a request() method, which is called with a WebOb Request object, and must return a WebOb Response object, like this:

class MyWidget(twc.Widget):
    id = 'my_widget'
    def request(cls, req):
        resp = webob.Response(request=req, content_type="text/html; charset=UTF8")
        # ...
        return resp

For the request() method to be called, the widget must be registered with the ControllersApp in the middleware. By default, the path is constructed from /controllers/, and the widget’s id. A request to /controllers/ refers to a widget with id index. You can specify controllers_prefix in the configuration.

For convenience, widgets that have a request() method, and an id will be registered automatically. By default, this uses a global ControllersApp instance, which is also the default controllers for make_middleware(). If you want to use multiple controller applications in a single python instance, you will need to override this.

You can also manually register widgets:

twc.core.register_controller(MyWidget, 'mywidget')

Sometimes it is useful to dynamically acquire what URL path a Widget’s controller is mounted on. For this you can use:


Methods to override

Instance method - get self and req. load from db
Class method - get cls and validated data
Return python data that is automatically converted to an ajax response


One of the main features of any forms library is the validation of form input, e.g checking that an email address is valid, or that a user name is not already taken. If there are validation errors, these must be displayed to the user in a helpful way. Many validation tasks are common, so these should be easy for the developer, while less-common tasks are still possible.

We can configure validation on form fields like this:

class child(twf.TableForm):
    name = twf.TextField(validator=twc.Required)
    group = twf.SingleSelectField(options=['', 'Red', 'Green', 'Blue'])
    notes = twf.TextArea(validator=twc.StringLengthValidator(min=10))

To enable validation we also need to modify the application to handle POST requests:

def app(environ, start_response):
    req = wo.Request(environ)
    resp = wo.Response(request=req, content_type="text/html; charset=UTF8")
    if req.method == 'GET':
        resp.body = MyForm.display().encode('utf-8')
    elif req.method == 'POST':
            data = MyForm.validate(req.POST)
            resp.body = 'Posted successfully ' + wo.html_escape(repr(data))
        except twc.ValidationError, e:
            resp.body = e.widget.display().encode('utf-8')
    return resp(environ, start_response)

If you submit the form with some invalid fields, you should see error messages sidle up to each relevant field.

Whole Form Message

If you want to display a message at the top of the form, when there are any errors, define the following validator:

class MyFormValidator(twc.Validator):
    msgs = {
        'childerror': ('form_childerror', 'There were problems with the details you entered. Review the messages below to correct your submission.'),

And in your form:

validator = MyFormValidator()


Validation is also responsible for conversion to and from python types. For example, the DateValidator takes a string from the form and produces a python date object. If it is unable to do this, that is a validation failure.

To keep related functionality together, validators also support coversion from python to string, for display. This should be complete, in that there are no python values that cause it to fail. It should also be precise, in that converting from python to string, and back again, should always give a value equal to the original python value. The converse is not always true, e.g. the string “1/2/2004” may be converted to a python date object, then back to “01/02/2004”.

Validation Errors

When there is an error, all fields should still be validated and multiple errors displayed, rather than stopping after the first error.

When validation fails, the user should see the invalid values they entered. This is helpful in the case that a field is entered only slightly wrong, e.g. a number entered as “2,000” when commas are not allowed. In such cases, conversion to and from python may not be possible, so the value is kept as a string. Some widgets will not be able to display an invalid value (e.g. selection fields); this is fine, they just have to do the best they can.

When there is an error is some fields, other valid fields can potentially normalise their value, by converting to python and back again (e.g. 01234 -> 1234). However, it was decided to use the original value in this case.

In some cases, validation may encounter a major error, as if the web user has tampered with the HTML source. However, we can never be completely sure this is the case, perhaps they have a buggy browser, or caught the site in the middle of an upgrade. In these cases, validation will produce the most helpful error messages it can, but not attempt to identify which field is at fault, nor redisplay invalid values.

Required Fields

If a field has no value, if defaults to None. It is down to that field’s validator to raise an error if the field is required. By default, fields are not required. It was considered to have a dedicated Missing class, but this was decided against, as None is already intended to convey the absence of data.

Security Consideration

When a widget is redisplayed after a validation failure, it’s value is derived from unvalidated user input. This means widgets must be “safe” for all input values. In practice, this is almost always the case without great care, so widgets are assumed to be safe.


If a particular widget is not safe in this way, it must override _validate() and set value to None in case of error.

Validation Messages

When validation fails, the validator raises ValidationError. This must be passed the short message name, e.g. “required”. Each validator has a dictionary mapping short names to messages that are presented to the user, e.g.:

msgs = {
    'tooshort': 'Value is too short',
    'toolong': 'Value is too long',

Messages can be overridden on a global basis, using validator_msgs on the middleware configuration. For example, the user may prefer “Value is required” instead of the default “Enter a value” for a missing field.

A Validator can also rename mesages, by specifying a tuple in the msgs dict. For example, ListLengthValidator is a subclass of LengthValidator which raises either tooshort or toolong. However, it’s desired to have different message names, so that any global override would be applied separately. The following msgs dict is used:

msgs = {
    'tooshort': ('list_tooshort', 'Select at least $min'),
    'toolong': ('list_toolong', 'Select no more than $max'),

Within the messages, tags like $min are substituted with the corresponding attribute from the validator. It is not possible to specify the value in this way; this is to discourage using values within messages.


Earlier versions of ToscaWidgets used FormEncode for validation and there are good reasons for this. Some aspects of the design work very well, and FormEncode has a lot of clever validators, e.g. the ability to check that a post code is in the correct format for a number of different countries.

However, there are challenges making FormEncode and ToscaWidgets work together. For example, both libraries store the widget hierarchy internally. This makes implementing some features (e.g. strip_name and tw2.dynforms.HidingSingleSelectField) difficult. There are different needs for the handling of unicode, leading ToscaWidgets to override some behaviour. Also, FormEncode just does not support client-side validation, a planned feature of ToscaWidgets 2.

ToscaWidgets 2 does not rely on FormEncode. However, developers can use FormEncode validators for individual fields. The API is compatible in that to_python() and from_python() are called for conversion and validation, and formencode.Invalid is caught. Also, if FormEncode is installed, the ValidationError class is a subclass of formencode.Invalid.

Using Validators

There’s two parts to using validators. First, specify validators in the widget definition, like this:

class RegisterUser(twf.TableForm):
    validator = twc.MatchValidator('email', 'confirm_email')
    name = twf.TextField()
    email = twf.TextField(validator=twc.EmailValidator)
    confirm_email = twf.PasswordField()

You can specify a validator on any widget, either a class or an instance. Using an instance lets you pass parameters to the validator. You can code your own validator by subclassing tw2.core.Validator. All validators have at least these parameters:

class tw2.core.Validator(**kw)[source]

Base class for validators

Whether empty values are forbidden in this field. (default: False)
Whether to strip leading and trailing space from the input, before any other validation. (default: True)

To convert and validate a value to Python, use the to_python() method, to convert back from Python, use from_python().

To create your own validators, sublass this class, and override any of _validate_python(), _convert_to_python(), or _convert_from_python(). Note that these methods are not meant to be used externally. All of them may raise ValidationErrors.

Second, when the form values are submitted, call validate() on the outermost widget. Pass this a dictionary of the request parameters. It will call the same method on all contained widgets, and either return the validated data, with all conversions applied, or raise tw2.core.ValidationError. In the case of a validation failure, it stores the invalid value and an error message on the affected widget.

Chaining Validators

In some cases you may want validation to succeed if any one of a number of checks pass. In other cases you may want validation to succeed only if the input passes all of a number of checks. For this, tw2.core provides the Any and All validators which are subclasses of the extendable CompoundValidator.


A two-pass approach is used internally, although this is generally hidden from the developer. When Widget.validate() is called it first calls:


This performs the first stage of validation. It takes a dictionary where some keys will be compound names, such as “form:subform:field” and converts this into a nested dict/list structure. It also performs unicode decoding, with the encoding specified in the middleware config.

If this fails, there is no attempt to determine which parameter failed; the whole submission is considered corrupt. If the root widget has an id, this is stripped from the dictionary, e.g. {'myid': {'param':'value', ...}} is converted to {'param':'value', ...}. A widget instance is created, and stored in request local storage. This allows compatibility with existing frameworks, e.g. the @validate decorator in TurboGears. There is a hook in display() that detects the request local instance. After creating the instance, validate works recursively, using the _validate().

Widget._validate(*args, **kw)[source]

Inner validation method; this is called by validate and should not be called directly. Overriding this method in widgets is discouraged; a custom validator should be coded instead. However, in some circumstances overriding is necessary.

RepeatingWidget._validate(*args, **kw)[source]

The value must either be a list or None. Each item in the list is passed to the corresponding child widget for validation. The resulting list is passed to this widget’s validator. If any of the child widgets produces a validation error, this widget generates a “childerror” failure.

CompoundWidget._validate(*args, **kw)[source]

The value must be a dict, or None. Each item in the dict is passed to the corresponding child widget for validation, with special consideration for _sub_compound widgets. If a child returns vd.EmptyField, that value is not included in the resulting dict at all, which is different to including None. Child widgets with a key are passed the validated value from the field the key references. The resulting dict is validated by this widget’s validator. If any child widgets produce an errors, this results in a “childerror” failure.

Both _validate() and to_python() take an optional state argument. CompoundWidget and RepeatingWidget pass the partially built dict/list to their child widgets as state. This is useful for creating validators like MatchValidator that reference sibling values. If one of the child widgets fails validation, the slot is filled with an Invalid instance.

General Considerations

Request-Local Storage

ToscaWidgets needs access to request-local storage. In particular, it’s important that the middleware sees the request-local information that was set when a widget is instatiated, so that resources are collected correctly.

The function tw2.core.request_local returns a dictionary that is local to the current request. Multiple calls in the same request always return the same dictionary. The default implementation of request_local is a thread-local system, which the middleware clears before and after each request.

In some situations thread-local is not appropriate, e.g. twisted. In this case the application will need to monkey patch request_local to use appropriate request_local storage.


tw2.core aims to take advantage of pkg_resources where it is available, but not to depend on it. This allows tw2.core to be used on Google App Engine. pkg_resources is used in two places:

  • In ResourcesApp, to serve resources from modules, which may be zipped eggs. If pkg_resources is not available, this uses a simpler system that does not support zipped eggs.
  • In EngingeManager, to load a templating engine from a text string, e.g. “genshi”. If pkg_resources is not available, this uses a simple, built-in mapping that covers the most common template engines.

Framework Interface

ToscaWidgets is designed to be standalone WSGI middeware and not have any framework interactions. However, when using ToscaWidgets with a framework, there are some configuration settings that need to be consistent with the framework, for correct interaction. Future vesions of ToscaWidgets may include framework-specific hooks to automatically gather this configuration. The settings are:

  • default_view - the template engine used by the framework. When root widgets are rendered, they will return a type suitable for including in this template engine. This setting is not needed if only Page widgets are used as root widgets, as there is no containing template in that case.
  • translator - needed for ToscaWidget to use the same i18n function as the framework.

Unit Tests

To run the tests, in tw2.devtools/tests issue:

nosetests --with-doctest --doctest-extension=.txt