Drawing Redux Form FieldArrays with Pug

Having been spoiled by the Rails Prime Stack for nearly a decade, in my React projects I prefer using Pug (née Jade) instead of JSX for its Haml-like syntax. A lot of people praise Pug and Haml for saving them typing, and while that’s nice, the real appeal to me is how easy they are to read. You spend a lot more time reading code than writing it, and Pug/Haml make the document structure immediately obvious. Making closing-tag errors obsolete is pretty nice, too.

With the babel-plugin-transform-react-pug package, you can replace your JSX with something like this:

class MyApp extends React.Component {
  render() {
    return pug`
      Provider(store=configureStore(this.props))
        table
          tbody
            tr
              td table
              td layout
              td anyone?
    `;
  }
}

But Pug is definitely not as widely adopted within React as Haml is within Rails, and it shows. I ran into a tricky issue using Pug to render FieldArrays with Redux Form and React Bootstrap. To combine those two packages I’m basically following John Bennett’s advice, except with Pug.

Here is how typical scalar fields work:

const renderInput = ({input, label, type, meta}) => pug`
  FormGroup(controlId=input.name validationState=${validState(meta)})
    Col(componentClass=${ControlLabel} sm=2)
      = label || humanizeString(input.name)
    Col(sm=5)
      FormControl(...input type=type)
`

class EditClientPage extends React.Component {
  render() {
    return pug`
      Form(horizontal onSubmit=${this.props.handleSubmit})
        fieldset
          Field(name='name' component=renderInput type='text')
          Field(name='paymentDue' component=renderInput type='number')
    `
  }
}

That’s from my time-tracking app where I wrote about using FieldArray with redux-form-validators. The Field component is from Redux Form, and everything else is from React-Bootstrap (Form, FormGroup, Col, ControlLabel, and FormControl). You can see that Field expects a custom component to draw its details, giving you total flexibility how you structure your form’s DOM. Most of the Boostrap components go inside the custom component used by Field.

So far that’s pretty nice, but if you have a FieldArray you need more nesting. A FieldArray is also part of Redux Form, and is used to draw a list of child records with their sub-fields. In my case I want the page to have one or more “work categories”, each with a name and hourly rate, e.g. “Design” and “Development”.

Like Field, a FieldArray delegates rendering to a custom component. Then that component will render the individual Fields (each with their own custom component in turn). If you adapted the Redux Form docs’ example, you might try something like this:

const renderSimpleInput = ({input, placeholder, type, meta}) => pug`
  span(class=${validClass(meta)})
    FormControl(...input placeholder=placeholder type=type)
`

const renderWorkCategories = ({fields, meta}) => `pug
  .noop
    = fields.map((wc, index) => `pug
      FormGroup(key=${index})
        Col(sm=2)
          Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
        Col(sm=5)
          Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
        Col(sm=2)
          a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
    `)
    FormGroup
      Col(smOffset=2 sm=5)
        a.btn.btn-default(onClick=${()=>fields.push({})}) add rate
`

class EditClientPage extends React.Component {
  render() {
    return pug`
      ...
      FieldArray(name='workCategories' component=${renderWorkCategories})
      ...
    `
  }
}

The problem is that you can’t nest pug strings like that. I’m not sure if the problem is with the Babel transformer or the pug parser itself, but you get an error. Of course that’s not idiomatic Pug anyway, but surprisingly, you can’t use Pug’s each command either:

const renderWorkCategories = ({fields, meta}) => `pug
  .noop
    each wc, index in fields
      FormGroup(key=${index})
        Col(sm=2)
          Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
        Col(sm=5)
          Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
        Col(sm=2)
          a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
    ...
`

This gives you the error Expected "fields" to be an array because it was passed to each. Apparently Redux Form is not using a normal array here, but its own special object.

The trick is to call getAll, like this:

each wc, index in fields.getAll()
  FormGroup(key=${index})
    Col(sm=2)
      Field(name=${`workCategories[${index}].name`} component=renderSimpleInput type='text' placeholder='name')
    Col(sm=5)
      Field(name=${`workCategories[${index}].rate`} component=renderSimpleInput type='number' placeholder='rate')
    Col(sm=2)
      a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`

Note that we also had to stop using ${wc} and are building the field name “by hand”. Personally I think we can stop here and be done, but if that feels like breaking encapsulation to you, or if you want something more generic that doesn’t need to “know” its FieldArray name, there is another way to do it. Even if it’s a bit too much for a real project, it’s interesting enough that it’s maybe worth seeing.

To start, we need to call fields.map with another custom component. This almost works:

const renderWorkCategory = (wc, index) => `pug
  FormGroup(key=${index})
    Col(sm=2)
      Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
    Col(sm=5)
      Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
    Col(sm=2)
      a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`

const renderWorkCategories = ({fields, meta}) => `pug
  .noop
    = fields.map(renderWorkCategory)
    ...

The only problem is the remove button: fields is no longer in scope!

The solution is to use currying. The component we hand to fields.map will be a partially-applied function, generated by passing in fields early. ES6 syntax makes it really easy. The full code looks like this:

const renderWorkCategory = (fields) => (wc, index) => `pug
  FormGroup(key=${index})
    Col(sm=2)
      Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
    Col(sm=5)
      Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
    Col(sm=2)
      a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`

const renderWorkCategories = ({fields, meta}) => `pug
  .noop
    = fields.map(renderWorkCategory(fields))
    ...
`

You may recall we also used currying to combine redux-form-validators with FieldArray. It can really come in handy!

As I’ve said before, programming one thing is usually easy; it gets hard when we try to do many things at once. Here I show how to use Pug with a FieldArray from Redux Form, on a page styled with React Bootstrap. I hope you found it useful if like me you are trying to have your cake and eat it too. :-)

UPDATE: It turns out I was making things too complicated: fields.map takes an optional third parameter, fields. That means there is no need to curry renderWorkCategory and pass in fields early. Instead of this:

const renderWorkCategory = (fields) => (wc, index) => ...

you can just say this:

const renderWorkCategory = (wc, index, fields) => ...

I guess it pays to read the documentation! :-)

Validating FieldArrays in Redux Form

I have a Redux Form project where I’m using redux-form-validators to get easy Rails-style validations. Its docs explain how to define validators for normal scalar fields, but I thought I’d add how to do the same for a FieldArray, which is a list of zero of more sub-fields.

In my case I’m working on a time-tracking and invoicing application, where admins can edit a client. The form has regular fields for the client’s name, the invoicing frequency, etc., and then it also has “work categories”. Each client has one or more work categories, and a work category is just a name and an hourly rate. For instance you might charge one rate for design and another for development, or you might track retainer hours at $0/hour and extra hours at your normal rate.

Redux Form makes it really easy to include child records right inside the main form using FieldArray. Their docs give a nice example of how to validate those nested fields, but it’s pretty DIY and verbose.

With redux-form-validators on the other hand, it’s easy. First you define a validation object with the rules for each field, like so:

const validations = {
  name: [required()],
  paymentDue: [required(), numericality({int: true, '>=': 0})],
  // ...
};

Then you write a little validation function to pass to Redux Form:

const validate = function(values) => {
  const errors = {}
  for (let field in validations) {
    let value = values[field]
    errors[field] = validations[field].map(validateField => {
      return validateField(value, values)
    }).find(x => x) // Take the first non-null error message.
  }
  return errors
};

export default connect(mapStateToProps, mapDispatchToProps)(
  reduxForm({
    form: 'editClient',
    validate
  })(EditClientPage)
);

That is just straight from the docs. For each field it iterates over its validators and reports the first error it finds. It’s simple, but it doesn’t know how to handle nesting like with FieldArray.

But notice you can really do whatever you like. It’s a bit striking for someone used to Rails how exposed and customizable everything is here. So how can we rewrite validate to be smarter?

Ideally we’d like to support validations both on individual sub-fields, like the name of a single work category, and also on the workCategories as a whole, for example checking that you have at least one. This is exactly what the Redux Form example does above: it checks that your club has at least one member, and that each member has a first and last name.

Because this is a FieldArray, Redux Form expects a different structure for its entry in the errors object returned by validate. Normally you’d have a string value, like this:

{
  paymentDue: 'must be numeric'
}

But for a FieldArray you want to pass an array with one error object for each corresponding FieldArray element, like this:

{
  workCategories: [
    {},   // no errors
    {name: 'is required'},
    {name: 'is too short', rate: 'must be numeric'},
  ]
}

Those sub-errors will get passed in the meta object given to the component used to render each Field. Again, you can see that happening in the Redux Form example.

In addition, the array may have its own _error attribute for top-level errors. That gets passed as meta.error to the FieldArray component itself. So _error is not just a convention; it’s a “magic” attribute built into Redux Form. We want to set it too.

I want a way to define all these validations in one big object: top-level fields, the FieldArray itself, and the fields of individual FieldArray records. Here is the structure I set up:

const validations = {
  name:       [required()],
  paymentDue: [required(), numericality({int: true, '>=': 0})],
  ...
  workCategories: {
    _error: [
      required({msg: "Please enter at least one work category."}),
      length({min: 1, msg: "Please enter at least one work category."})
    ],
    name:   [required()],
    rate:   [required(), numericality({'>=': 0})],
  },
};

Then instead of the recommended validate function I used this:

const buildErrors = (validations) => (values) => {
  const errors = {};
  for (let field in validations) {
    if (field === '_error') continue;
    let value = values[field];
    const fieldValidations = validations[field];
    if (fieldValidations.constructor === Array) {
      errors[field] = fieldValidations
          .map(validateField => validateField(value, values))
          .find(x => x);
    } else {
      errors[field] = value ? value.map(o => buildErrors(fieldValidations)(o)) : [];
      if (fieldValidations._error) {
        errors[field]._error = fieldValidations._error
            .map(validateField => validateField(value, values))
            .find(x => x);
    }
  }
  return errors;
}

export default connect(mapStateToProps, mapDispatchToProps)(
  reduxForm({
    form: 'editClient',
    validate: buildErrors(validations),
  })(EditClientPage)
);

There are a few things to note here: I’m using a recursive function, where each call handles one “level” of the validations object. It iterates over the field names, and if the field has an array, it handles it as before. Otherwise it expects a nested object structured just like the outermost object, which each sub-field has its own array of validators. There may also be a list of validators under _errors, and those are handled specially. I’m using currying so that I can build a top-level validation function for reduxForm as well as nested functions for each FieldArray.

This function also supports reporting errors both on sub-fields and the whole FieldArray at the same time. That wouldn’t happen if the only FieldArray check was to have at least one element, like here, but you can imagine scenarios where you want to validate the FieldArray as a whole even when it isn’t empty.

I’m happy that this approach lets me combine the ease of redux-form-validators with FieldArray to get the best of both worlds. I also like that buildErrors is general enough that I can move it to a utility collection, and not write it separately for each form in my app.

Also: you might enjoy my follow-up article showing how to render the form Fields and FieldArrays with Pug and Bootstrap.

Testing Your ActionMailer Configuration

Sometimes with Rails you just want to test basic email sending from the console. Here is how you can do it, without needing a Mailer or views or anything:

ActionMailer::Base.mail(
  to:      'dest@example.com',
  from:    'src@example.com', 
  subject: 'test',
  body:    'testing',
).deliver_now

Or in older versions use deliver instead of deliver_now. This will use your ActionMailer settings but doesn’t require anything else, so you can test just the SMTP config (or whatever).

Adding an if method to ActiveRecord::Relation

I often find myself chaining ActiveRecord scopes, like this:

q = Article
  .not_deleted
  .published_within(start_date, end_date)
q = q.with_tags(tags) if tags.any?

I wish there were a nicer way to put in the conditional scopes, without assigning to a temporary variable. What I really want is to write this:

Article
  .not_deleted
  .published_within(start_date, end_date)
  .if(tags.any?) { with_tags(tags) }

Wouldn’t that be nicer?

Many years ago I brought this up on the pdxruby mailing list, and no one seemed very interested, but I’ve always wanted it in my projects.

Here is a naïve implementation, which just for simplicity I’ll add to Object instead of ActiveRecord::Relation (where it really belongs):

class Object

  def if(condition, &block)
    condition ? instance_eval(&block) : self
  end

end

5.if(true)  { self + 7 }    # equals 12
5.if(false) { self + 7 }    # equals 5

That almost works! The problem is that inside the block, things don’t really act like a closure. If we used this implementation, our example above would give a NameError about not finding a tags method or local variable. That’s because everything inside the block is evaluated with self set to the ActiveRecord::Relation instance.

Fortunately there is a way to fix it! Before calling the block, we can save the outside self, and then we can use method_missing to delegate any failures to there.

There is a nice writeup of this “cloaking” trick if you want more details. But if you read that article, perhaps you will notice it is not thread-safe, because it temporarily adds a method to the class, and two threads could stomp on each other if they did that at the same time.

This approach was in Rails ActiveSupport for a while as Proc#bind. They even fixed the multi-threading problem (more or less . . .) by generating a different method name every time. Unfortunately that created a new problem: since define_method takes a symbol, this creates more and more symbols, which in Ruby are never garbage collected! Effectively this is a memory leak. Eventually the Rails team deprecated it.

But we can still add something similar that doesn’t leak memory and is thread-safe. We just have to protect the brief moment when we define the method and then remove it, which is simple with a Mutex. In theory taking a lock adds some overhead, and possibly contention, but we don’t expect this to be a “hot spot”, so in practice the contention should be zero and the overhead trivial.

And here is our implementation (not on Object any more):

module ActiveRecord
  class Relation

    CLOAKER_MUTEX = Mutex.new

    def if(condition, &block)
      if condition
        meth = self.class.class_eval do
          CLOAKER_MUTEX.synchronize do
            define_method :cloaker_, &block
            meth = instance_method :cloaker_
            remove_method :cloaker_
            meth
          end
        end
        with_previous_context(block.binding) { meth.bind(self).call }
      else
        self
      end
    end

    def with_previous_context(binding, &block)
      @previous_context = binding.eval('self')
      result = block.call
      @previous_context = nil
      result
    end

    def method_missing(method, *args, &block)
      super
    rescue NameError => e
      if @previous_context
        @previous_context.send(method, *args, &block)
      else
        raise e
      end
    end
  end
end

Put that in config/initializers and try it out!

An nginx HTTP-to-HTTPS Redirect Mystery, and Configuration Advice

I noticed a weird thing last night on an nginx server I administer. The logs were full of lines like this:

42.232.104.114 - - [25/Mar/2018:04:50:49 +0000] "GET http://www.ioffer.com/i/new-fashion-fine-gold-bracelet-versaec-bracelet-641175733 HTTP/1.1" 301 185 "http://www.ioffer.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Hotbar 4.1.8.0; RogueCleaner; Alexa Toolbar)"

Traffic was streaming in continuously: maybe ten or twenty requests per second.

At first I thought the server had been hacked, but really it seemed people were just sending lots of traffic and getting 301 redirects. I could reproduce the problem with a telnet session:

$ telnet 198.51.100.42 80
Trying 198.51.100.42...
Connected to example.com.
Escape character is '^]'.
GET http://www.ioffer.com/i/men-shouder-messenger-bag-briefcase-shoulder-bags-649303549 HTTP/1.1
Host: www.ioffer.com

HTTP/1.1 301 Moved Permanently
Server: nginx/1.10.1
Date: Sun, 25 Mar 2018 04:56:06 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.ioffer.com/i/men-shouder-messenger-bag-briefcase-shoulder-bags-649303549

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.10.1</center>
</body>
</html>

In that session, I typed the first two lines after Escape character..., plus the blank line following. Normally a browser would not include a whole URL after GET, only the path, like GET /about.html HTTP/1.1, but including the whole URL is used when going through a proxy. Also it may be possible to leave off the Host header. Technically it is required for HTTP/1.1, so I added it just out of habit. I didn’t test without it.

So what was happening here? I was following some common advice to redirect HTTP to HTTPS, using configuration like this:

server {
  listen 80;
  server_name example.com *.example.com;
  return 301 https://$host$request_uri;
}

The problem is the $host evaluates to whatever the browser wants. In order of precedence, it can be (1) the host name from the request line (as in my example), (2) the Host header, or (3) what you declared as the server_name for the matching block. A safer alternative is to send people to https://$server_name$request_uri. Then everything is under your control. You can see people recommending that on the ServerFault page.

The problem is when you declare more than one server_name, or when one of them is a wildcard. The $server_name variable always evaluates to the first one. It also doesn’t expand wildcards. (How could it?) That wouldn’t work for me, because in this project admins can add new subdomains any time, and I don’t want to update nginx config files when that happens.

Eventually I solved it using a config like this:

server {
  listen 80 default_server;
  server_name example.com;
  return 301 https://example.com$request_uri;
}
server {
  listen 80;
  server_name *.example.com;
  return 301 https://$host$request_uri;
}

Notice the default_server modifier. If any traffic actually matches *.example.com, it will use the second block, but otherwise it will fall back to the first block, where there is no $host variable, but just a hardcoded redirect to my own domain. After I made this change, I immediately saw traffic getting the redirect and making a second request back to my own machine, usually getting a 404. I expect pretty soon whoever is sending this traffic will knock it off. If not, I guess it’s free traffic for me. :-)

(Technically default_server is not required since if no block is the declared default, nginx will make the first the default automatically, but being explicit seems like an improvement, especially here where it matters.)

I believe I could also use a setup like this:

server {
  listen 80 default_server;
  return 301 "https://www.youtube.com/watch?v=dQw4w9WgXcQ";
}
server {
  listen 80;
  server_name example.com *.example.com;
  return 301 https://$host$request_uri;
}

There I list all my legitimate domains in the second block, so the default only matches when people are playing games. I guess I’m too nice to do that for real though, and anyway it would make me nervous that some other misconfiguration would activate that first block more often than I intended.

I’d still like to know what the point of this abuse was. My server wasn’t acting as an open proxy exactly, because it wasn’t fulfilling these requests on behalf of the clients and passing along the response (confirmed with tcpdump -n 'tcp[tcpflags] & (tcp-syn) != 0 and src host 198.51.100.42'); it was just sending a redirect. So what was it accomplishing?

The requests were for only a handful of different domains, mostly Chinese. They came from a variety of IPs. Sometimes an IP would make requests for hours and then disappear. The referrers varied. Most were normal, like Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0), but some mentioned toolbars like the example above.

I guess if it were DNS sending them to my server there would (possibly) be a redirect loop, which I wasn’t seeing. So was my server configured as their proxy?

To learn a little more, I moved nginx over to port 81 and ran this:

mkfifo reply
netcat -kl 80 < reply | tee saved | netcat 127.0.0.1 81 > reply

(At first instead of netcat I tried ./mitmproxy --save-stream-file +http.log --listen-port 80 --mode reverse:http://127.0.0.1:81 --set keep_host_header, but it threw errors on requests with full URLs (GET http://example.com/ HTTP/1.1) because it thought it should only see those in regular mode.)

Once netcat was running I could tail -F saved in another session. I saw requests like this:

GET http://www.douban.com/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Accept: text/html, */*
Accept-Language: zh-cn; en-us
Referer: http://www.douban.com/
Host: www.douban.com
Pragma: no-cache

I also saw one of these:

CONNECT www.kuaishou.com:443 HTTP/1.0
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11
Host: www.kuaishou.com:443
Content-Length: 0
Proxy-Connection: Keep-Alive
Pragma: no-cache

That is a more normal proxy request, although it seems like it was just regular scanning, because I’ve always returned a 400 to those.

Maybe the requests that were getting 301’d were just regular vulnerability scanning too? I don’t know. I seemed like something more specific than that.

The negatives for me were noisy logs and elevated bandwidth/CPU. Not a huge deal, but whatever was going on, I didn’t want to be a part of it.

. . .

By the way, as long as we’re talking about redirecting HTTP to HTTPS, I should mention HSTS, which is a way of telling browsers never to use HTTP here in the future. If you’re doing a redirect like this, it may be a good thing to add (to the HTTPS response, not the HTTP one). On the other hand it has some risks, if in the future you ever want to use HTTP again.

Counting Topologically Distinct Directed Acyclic Graphs with Marshmallows

I wrote a miniature Ruby gem to topologically sort a Directed Acyclic Graph (DAG), which is useful when you have a bunch of things that depend on each other (e.g. tasks), and you want to put them in a linear order.

Writing the test suite got me thinking about how to find all the topologically distinct directed acyclic graphs with number of vertices V and edges E. My current algorithm goes like this:

1. Start with some large number n of toothpicks and marshmallows.

2. Call the children.

3. Try to finish before all the marshmallows are gone.

Here is what we ended up with for all graphs of V = 4:

It’s not bad I think, but is a method known that works even without children? Are there any graphs I missed?

(Full disclosure: I redid this photo a couple days later with better-colored toothpicks, so now you can tell which way they point. Marshmallows may be crunchier than they appear.)

Next: Temporal Databases Annotated Bibliography