Having been spoiled by the Rails Prime Stack for nearly a decade, in my React projects I prefer using Pug (née Jade) instead of JSX for its Haml-like syntax. A lot of people praise Pug and Haml for saving them typing, and while that’s nice, the real appeal to me is how easy they are to read. You spend a lot more time reading code than writing it, and Pug/Haml make the document structure immediately obvious. Making closing-tag errors obsolete is pretty nice, too.
With the babel-plugin-transform-react-pug
package, you can replace your JSX with something like this:
class MyApp extends React.Component {
render() {
return pug`
Provider(store=configureStore(this.props))
table
tbody
tr
td table
td layout
td anyone?
`;
}
}
But Pug is definitely not as widely adopted within React as Haml is within Rails, and it shows. I ran into a tricky issue using Pug to render FieldArray
s with Redux Form and React Bootstrap. To combine those two packages I’m basically following John Bennett’s advice, except with Pug.
Here is how typical scalar fields work:
const renderInput = ({input, label, type, meta}) => pug`
FormGroup(controlId=input.name validationState=${validState(meta)})
Col(componentClass=${ControlLabel} sm=2)
= label || humanizeString(input.name)
Col(sm=5)
FormControl(...input type=type)
`
class EditClientPage extends React.Component {
render() {
return pug`
Form(horizontal onSubmit=${this.props.handleSubmit})
fieldset
Field(name='name' component=renderInput type='text')
Field(name='paymentDue' component=renderInput type='number')
`
}
}
That’s from my time-tracking app where I wrote about using FieldArray
with redux-form-validators
. The Field
component is from Redux Form, and everything else is from React-Bootstrap (Form
, FormGroup
, Col
, ControlLabel
, and FormControl
). You can see that Field
expects a custom component to draw its details, giving you total flexibility how you structure your form’s DOM. Most of the Boostrap components go inside the custom component used by Field
.
So far that’s pretty nice, but if you have a FieldArray
you need more nesting. A FieldArray
is also part of Redux Form, and is used to draw a list of child records with their sub-fields. In my case I want the page to have one or more “work categories”, each with a name and hourly rate, e.g. “Design” and “Development”.
Like Field
, a FieldArray
delegates rendering to a custom component. Then that component will render the individual Field
s (each with their own custom component in turn). If you adapted the Redux Form docs’ example, you might try something like this:
const renderSimpleInput = ({input, placeholder, type, meta}) => pug`
span(class=${validClass(meta)})
FormControl(...input placeholder=placeholder type=type)
`
const renderWorkCategories = ({fields, meta}) => `pug
.noop
= fields.map((wc, index) => `pug
FormGroup(key=${index})
Col(sm=2)
Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
Col(sm=5)
Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
Col(sm=2)
a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`)
FormGroup
Col(smOffset=2 sm=5)
a.btn.btn-default(onClick=${()=>fields.push({})}) add rate
`
class EditClientPage extends React.Component {
render() {
return pug`
...
FieldArray(name='workCategories' component=${renderWorkCategories})
...
`
}
}
The problem is that you can’t nest pug strings like that. I’m not sure if the problem is with the Babel transformer or the pug parser itself, but you get an error. Of course that’s not idiomatic Pug anyway, but surprisingly, you can’t use Pug’s each
command either:
const renderWorkCategories = ({fields, meta}) => `pug
.noop
each wc, index in fields
FormGroup(key=${index})
Col(sm=2)
Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
Col(sm=5)
Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
Col(sm=2)
a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
...
`
This gives you the error Expected "fields" to be an array because it was passed to each
. Apparently Redux Form is not using a normal array here, but its own special object.
The trick is to call getAll
, like this:
each wc, index in fields.getAll()
FormGroup(key=${index})
Col(sm=2)
Field(name=${`workCategories[${index}].name`} component=renderSimpleInput type='text' placeholder='name')
Col(sm=5)
Field(name=${`workCategories[${index}].rate`} component=renderSimpleInput type='number' placeholder='rate')
Col(sm=2)
a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`
Note that we also had to stop using ${wc}
and are building the field name “by hand”. Personally I think we can stop here and be done, but if that feels like breaking encapsulation to you, or if you want something more generic that doesn’t need to “know” its FieldArray
name, there is another way to do it. Even if it’s a bit too much for a real project, it’s interesting enough that it’s maybe worth seeing.
To start, we need to call fields.map
with another custom component. This almost works:
const renderWorkCategory = (wc, index) => `pug
FormGroup(key=${index})
Col(sm=2)
Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
Col(sm=5)
Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
Col(sm=2)
a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`
const renderWorkCategories = ({fields, meta}) => `pug
.noop
= fields.map(renderWorkCategory)
...
The only problem is the remove button: fields
is no longer in scope!
The solution is to use currying. The component we hand to fields.map
will be a partially-applied function, generated by passing in fields
early. ES6 syntax makes it really easy. The full code looks like this:
const renderWorkCategory = (fields) => (wc, index) => `pug
FormGroup(key=${index})
Col(sm=2)
Field(name=${`${wc}.name`} component=renderSimpleInput type='text' placeholder='name')
Col(sm=5)
Field(name=${`${wc}.rate`} component=renderSimpleInput type='number' placeholder='rate')
Col(sm=2)
a.btn.btn-default(onClick=${()=>fields.remove(index)}) remove
`
const renderWorkCategories = ({fields, meta}) => `pug
.noop
= fields.map(renderWorkCategory(fields))
...
`
You may recall we also used currying to combine redux-form-validators
with FieldArray
. It can really come in handy!
As I’ve said before, programming one thing is usually easy; it gets hard when we try to do many things at once. Here I show how to use Pug with a FieldArray
from Redux Form, on a page styled with React Bootstrap. I hope you found it useful if like me you are trying to have your cake and eat it too. :-)
UPDATE: It turns out I was making things too complicated: fields.map
takes an optional third parameter, fields
. That means there is no need to curry renderWorkCategory
and pass in fields
early. Instead of this:
const renderWorkCategory = (fields) => (wc, index) => ...
you can just say this:
const renderWorkCategory = (wc, index, fields) => ...
I guess it pays to read the documentation! :-)
I have a Redux Form project where I’m using redux-form-validators
to get easy Rails-style validations. Its docs explain how to define validators for normal scalar fields, but I thought I’d add how to do the same for a FieldArray
, which is a list of zero of more sub-fields.
In my case I’m working on a time-tracking and invoicing application, where admins can edit a client. The form has regular fields for the client’s name, the invoicing frequency, etc., and then it also has “work categories”. Each client has one or more work categories, and a work category is just a name and an hourly rate. For instance you might charge one rate for design and another for development, or you might track retainer hours at $0/hour and extra hours at your normal rate.
Redux Form makes it really easy to include child records right inside the main form using FieldArray
. Their docs give a nice example of how to validate those nested fields, but it’s pretty DIY and verbose.
With redux-form-validators
on the other hand, it’s easy. First you define a validation object with the rules for each field, like so:
const validations = {
name: [required()],
paymentDue: [required(), numericality({int: true, '>=': 0})],
// ...
};
Then you write a little validation function to pass to Redux Form:
const validate = function(values) => {
const errors = {}
for (let field in validations) {
let value = values[field]
errors[field] = validations[field].map(validateField => {
return validateField(value, values)
}).find(x => x) // Take the first non-null error message.
}
return errors
};
export default connect(mapStateToProps, mapDispatchToProps)(
reduxForm({
form: 'editClient',
validate
})(EditClientPage)
);
That is just straight from the docs. For each field it iterates over its validators and reports the first error it finds. It’s simple, but it doesn’t know how to handle nesting like with FieldArray
.
But notice you can really do whatever you like. It’s a bit striking for someone used to Rails how exposed and customizable everything is here. So how can we rewrite validate
to be smarter?
Ideally we’d like to support validations both on individual sub-fields, like the name
of a single work category, and also on the workCategories
as a whole, for example checking that you have at least one. This is exactly what the Redux Form example does above: it checks that your club has at least one member, and that each member has a first and last name.
Because this is a FieldArray
, Redux Form expects a different structure for its entry in the errors object returned by validate
. Normally you’d have a string value, like this:
{
paymentDue: 'must be numeric'
}
But for a FieldArray
you want to pass an array with one error object for each corresponding FieldArray
element, like this:
{
workCategories: [
{}, // no errors
{name: 'is required'},
{name: 'is too short', rate: 'must be numeric'},
]
}
Those sub-errors will get passed in the meta
object given to the component used to render each Field
. Again, you can see that happening in the Redux Form example.
In addition, the array may have its own _error
attribute for top-level errors. That gets passed as meta.error
to the FieldArray
component itself. So _error
is not just a convention; it’s a “magic” attribute built into Redux Form. We want to set it too.
I want a way to define all these validations in one big object: top-level fields, the FieldArray
itself, and the fields of individual FieldArray
records. Here is the structure I set up:
const validations = {
name: [required()],
paymentDue: [required(), numericality({int: true, '>=': 0})],
...
workCategories: {
_error: [
required({msg: "Please enter at least one work category."}),
length({min: 1, msg: "Please enter at least one work category."})
],
name: [required()],
rate: [required(), numericality({'>=': 0})],
},
};
Then instead of the recommended validate
function I used this:
const buildErrors = (validations) => (values) => {
const errors = {};
for (let field in validations) {
if (field === '_error') continue;
let value = values[field];
const fieldValidations = validations[field];
if (fieldValidations.constructor === Array) {
errors[field] = fieldValidations
.map(validateField => validateField(value, values))
.find(x => x);
} else {
errors[field] = value ? value.map(o => buildErrors(fieldValidations)(o)) : [];
if (fieldValidations._error) {
errors[field]._error = fieldValidations._error
.map(validateField => validateField(value, values))
.find(x => x);
}
}
return errors;
}
export default connect(mapStateToProps, mapDispatchToProps)(
reduxForm({
form: 'editClient',
validate: buildErrors(validations),
})(EditClientPage)
);
There are a few things to note here: I’m using a recursive function, where each call handles one “level” of the validations
object. It iterates over the field names, and if the field has an array, it handles it as before. Otherwise it expects a nested object structured just like the outermost object, which each sub-field has its own array of validators. There may also be a list of validators under _errors
, and those are handled specially. I’m using currying so that I can build a top-level validation function for reduxForm
as well as nested functions for each FieldArray
.
This function also supports reporting errors both on sub-fields and the whole FieldArray
at the same time. That wouldn’t happen if the only FieldArray
check was to have at least one element, like here, but you can imagine scenarios where you want to validate the FieldArray
as a whole even when it isn’t empty.
I’m happy that this approach lets me combine the ease of redux-form-validators
with FieldArray
to get the best of both worlds. I also like that buildErrors
is general enough that I can move it to a utility collection, and not write it separately for each form in my app.
Also: you might enjoy my follow-up article showing how to render the form Field
s and FieldArray
s with Pug and Bootstrap.
Sometimes with Rails you just want to test basic email sending from the console. Here is how you can do it, without needing a Mailer or views or anything:
ActionMailer::Base.mail(
to: 'dest@example.com',
from: 'src@example.com',
subject: 'test',
body: 'testing',
).deliver_now
Or in older versions use deliver
instead of deliver_now
. This will use your ActionMailer settings but doesn’t require anything else, so you can test just the SMTP config (or whatever).
I often find myself chaining ActiveRecord scopes, like this:
q = Article
.not_deleted
.published_within(start_date, end_date)
q = q.with_tags(tags) if tags.any?
I wish there were a nicer way to put in the conditional scopes, without assigning to a temporary variable. What I really want is to write this:
Article
.not_deleted
.published_within(start_date, end_date)
.if(tags.any?) { with_tags(tags) }
Wouldn’t that be nicer?
Many years ago I brought this up on the pdxruby mailing list, and no one seemed very interested, but I’ve always wanted it in my projects.
Here is a naïve implementation, which just for simplicity I’ll add to Object
instead of ActiveRecord::Relation
(where it really belongs):
class Object
def if(condition, &block)
condition ? instance_eval(&block) : self
end
end
5.if(true) { self + 7 } # equals 12
5.if(false) { self + 7 } # equals 5
That almost works! The problem is that inside the block, things don’t really act like a closure. If we used this implementation, our example above would give a NameError
about not finding a tags
method or local variable. That’s because everything inside the block is evaluated with self
set to the ActiveRecord::Relation
instance.
Fortunately there is a way to fix it! Before calling the block, we can save the outside self
, and then we can use method_missing
to delegate any failures to there.
There is a nice writeup of this “cloaking” trick if you want more details. But if you read that article, perhaps you will notice it is not thread-safe, because it temporarily adds a method to the class, and two threads could stomp on each other if they did that at the same time.
This approach was in Rails ActiveSupport for a while as Proc#bind
. They even fixed the multi-threading problem (more or less . . .) by generating a different method name every time. Unfortunately that created a new problem: since define_method
takes a symbol, this creates more and more symbols, which in Ruby are never garbage collected! Effectively this is a memory leak. Eventually the Rails team deprecated it.
But we can still add something similar that doesn’t leak memory and is thread-safe. We just have to protect the brief moment when we define the method and then remove it, which is simple with a Mutex
. In theory taking a lock adds some overhead, and possibly contention, but we don’t expect this to be a “hot spot”, so in practice the contention should be zero and the overhead trivial.
And here is our implementation (not on Object
any more):
module ActiveRecord
class Relation
CLOAKER_MUTEX = Mutex.new
def if(condition, &block)
if condition
meth = self.class.class_eval do
CLOAKER_MUTEX.synchronize do
define_method :cloaker_, &block
meth = instance_method :cloaker_
remove_method :cloaker_
meth
end
end
with_previous_context(block.binding) { meth.bind(self).call }
else
self
end
end
def with_previous_context(binding, &block)
@previous_context = binding.eval('self')
result = block.call
@previous_context = nil
result
end
def method_missing(method, *args, &block)
super
rescue NameError => e
if @previous_context
@previous_context.send(method, *args, &block)
else
raise e
end
end
end
end
Put that in config/initializers
and try it out!
I noticed a weird thing last night on an nginx server I administer. The logs were full of lines like this:
42.232.104.114 - - [25/Mar/2018:04:50:49 +0000] "GET http://www.ioffer.com/i/new-fashion-fine-gold-bracelet-versaec-bracelet-641175733 HTTP/1.1" 301 185 "http://www.ioffer.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Hotbar 4.1.8.0; RogueCleaner; Alexa Toolbar)"
Traffic was streaming in continuously: maybe ten or twenty requests per second.
At first I thought the server had been hacked, but really it seemed people were just sending lots of traffic and getting 301 redirects. I could reproduce the problem with a telnet session:
$ telnet 198.51.100.42 80
Trying 198.51.100.42...
Connected to example.com.
Escape character is '^]'.
GET http://www.ioffer.com/i/men-shouder-messenger-bag-briefcase-shoulder-bags-649303549 HTTP/1.1
Host: www.ioffer.com
HTTP/1.1 301 Moved Permanently
Server: nginx/1.10.1
Date: Sun, 25 Mar 2018 04:56:06 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.ioffer.com/i/men-shouder-messenger-bag-briefcase-shoulder-bags-649303549
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.10.1</center>
</body>
</html>
In that session, I typed the first two lines after Escape character...
, plus the blank line following. Normally a browser would not include a whole URL after GET
, only the path, like GET /about.html HTTP/1.1
, but including the whole URL is used when going through a proxy. Also it may be possible to leave off the Host
header. Technically it is required for HTTP/1.1, so I added it just out of habit. I didn’t test without it.
So what was happening here? I was following some common advice to redirect HTTP to HTTPS, using configuration like this:
server {
listen 80;
server_name example.com *.example.com;
return 301 https://$host$request_uri;
}
The problem is the $host
evaluates to whatever the browser wants. In order of precedence, it can be (1) the host name from the request line (as in my example), (2) the Host
header, or (3) what you declared as the server_name
for the matching block. A safer alternative is to send people to https://$server_name$request_uri
. Then everything is under your control. You can see people recommending that on the ServerFault page.
The problem is when you declare more than one server_name
, or when one of them is a wildcard. The $server_name
variable always evaluates to the first one. It also doesn’t expand wildcards. (How could it?) That wouldn’t work for me, because in this project admins can add new subdomains any time, and I don’t want to update nginx config files when that happens.
Eventually I solved it using a config like this:
server {
listen 80 default_server;
server_name example.com;
return 301 https://example.com$request_uri;
}
server {
listen 80;
server_name *.example.com;
return 301 https://$host$request_uri;
}
Notice the default_server
modifier. If any traffic actually matches *.example.com
, it will use the second block, but otherwise it will fall back to the first block, where there is no $host
variable, but just a hardcoded redirect to my own domain. After I made this change, I immediately saw traffic getting the redirect and making a second request back to my own machine, usually getting a 404. I expect pretty soon whoever is sending this traffic will knock it off. If not, I guess it’s free traffic for me. :-)
(Technically default_server
is not required since if no block is the declared default, nginx will make the first the default automatically, but being explicit seems like an improvement, especially here where it matters.)
I believe I could also use a setup like this:
server {
listen 80 default_server;
return 301 "https://www.youtube.com/watch?v=dQw4w9WgXcQ";
}
server {
listen 80;
server_name example.com *.example.com;
return 301 https://$host$request_uri;
}
There I list all my legitimate domains in the second block, so the default only matches when people are playing games. I guess I’m too nice to do that for real though, and anyway it would make me nervous that some other misconfiguration would activate that first block more often than I intended.
I’d still like to know what the point of this abuse was. My server wasn’t acting as an open proxy exactly, because it wasn’t fulfilling these requests on behalf of the clients and passing along the response (confirmed with tcpdump -n 'tcp[tcpflags] & (tcp-syn) != 0 and src host 198.51.100.42'
); it was just sending a redirect. So what was it accomplishing?
The requests were for only a handful of different domains, mostly Chinese. They came from a variety of IPs. Sometimes an IP would make requests for hours and then disappear. The referrers varied. Most were normal, like Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
, but some mentioned toolbars like the example above.
I guess if it were DNS sending them to my server there would (possibly) be a redirect loop, which I wasn’t seeing. So was my server configured as their proxy?
To learn a little more, I moved nginx over to port 81 and ran this:
mkfifo reply
netcat -kl 80 < reply | tee saved | netcat 127.0.0.1 81 > reply
(At first instead of netcat
I tried ./mitmproxy --save-stream-file +http.log --listen-port 80 --mode reverse:http://127.0.0.1:81 --set keep_host_header
, but it threw errors on requests with full URLs (GET http://example.com/ HTTP/1.1
) because it thought it should only see those in regular mode.)
Once netcat was running I could tail -F saved
in another session. I saw requests like this:
GET http://www.douban.com/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Accept: text/html, */*
Accept-Language: zh-cn; en-us
Referer: http://www.douban.com/
Host: www.douban.com
Pragma: no-cache
I also saw one of these:
CONNECT www.kuaishou.com:443 HTTP/1.0
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11
Host: www.kuaishou.com:443
Content-Length: 0
Proxy-Connection: Keep-Alive
Pragma: no-cache
That is a more normal proxy request, although it seems like it was just regular scanning, because I’ve always returned a 400 to those.
Maybe the requests that were getting 301’d were just regular vulnerability scanning too? I don’t know. I seemed like something more specific than that.
The negatives for me were noisy logs and elevated bandwidth/CPU. Not a huge deal, but whatever was going on, I didn’t want to be a part of it.
. . .
By the way, as long as we’re talking about redirecting HTTP to HTTPS, I should mention HSTS, which is a way of telling browsers never to use HTTP here in the future. If you’re doing a redirect like this, it may be a good thing to add (to the HTTPS response, not the HTTP one). On the other hand it has some risks, if in the future you ever want to use HTTP again.
I wrote a miniature Ruby gem to topologically sort a Directed Acyclic Graph (DAG), which is useful when you have a bunch of things that depend on each other (e.g. tasks), and you want to put them in a linear order.
Writing the test suite got me thinking about how to find all the topologically distinct directed acyclic graphs with number of vertices V
and edges E
. My current algorithm goes like this:
Start with some large number n
of toothpicks and marshmallows.
Call the children.
Try to finish before all the marshmallows are gone.
Here is what we ended up with for all graphs of V = 4
:
It’s not bad I think, but is a method known that works even without children? Are there any graphs I missed?
(Full disclosure: I redid this photo a couple days later with better-colored toothpicks, so now you can tell which way they point. Marshmallows may be crunchier than they appear.)
Next: Temporal Databases Annotated Bibliography