Category Archives: Musing

Some Holiday Cheer for Nascent Writers

Readers of my blog know that I’m not given to chatty, optimistic posts. In fact, my typical post is more along the lines of “Not only will you fail at writing, but your cat will run away, your house will burn down, and you’ll spend the rest of your life tweeting from a phone you forgot to take out of airplane mode.” This post is different. While it isn’t quite optimistic, it does offer a perspective you may find uplifting, perhaps even liberating.

I’ve participated in many writing groups over the years and have managed a few as well, including one which technically qualified as Boston’s largest at the time. I have many writer acquaintances and even a few writer friends. I’m not spouting this to toot my own horn, but to lend credence to what I am about to say.

Many writers seem to have a notion of success which I deem unhealthy. I’m not saying that we should redefine "success" so that everybody is a winner or any such happy horseshit. The problem is that writers have two competing, and largely incompatible, goals. I speak here of real writers, not people who simply produce a product. The difference, to my mind, is that a writer wants to be proud of their work. My own standard is that I write what I want to read, and I think many of us implicitly or explicitly have a similar benchmark. We may try to cater to the taste of the crowd or steer toward marketability, but catering is different than pandering and steering is different than veering. At the end of the day, the stories and books which we produce must satisfy us first and foremost. This does not mean we always succeed in meeting that standard, but it is what we strive to aesthetically achieve.

However, American culture imbues us with another standard of success — one that is financial and social. Though success in this regard can be achieved via various avenues, the essential value it embodies — and that which our society most greatly respects — is the ability to sell things. In practice, this often takes the form of selling people stuff they don’t really want — but it need not. We are taught that the "net worth" of an individual is the sum of their possessions, marked to market, and distilled to a number. We are taught that we can order people by importance from lowest to greatest based on that number. We are taught that if one author sells many books and another sells few, the first is much "better" than the second. And we are taught that if a big publisher picks up a book, it is a "better" book than any which are self-published. While many of us may vociferously reject such a simplistic and materialistic outlook, we nonetheless are thralls to it. We may know that fatty foods are bad for us and that consumerism destroys the environment, yet there we all are in front of brand spanking new 100-inch televisions with bags of Fritos in our hands. Knowing and feeling are two different things, as are knowing and doing. We know we shouldn’t adopt the typical American view of success but we do anyway. Understanding and accepting that we are susceptible to such internal contradictions is crucial to avoiding the misery they otherwise can engender.

For a writer, the ability to sell our writing is essential for American-style success. These days, this entails also selling our "own story" as well. I personally find this obsession with the author rather than their work vapid at best and venal at worst, but it’s a fact of the market. The demographics of who reads and how are vastly different than a few years ago, as are the nature of publishers and what they seek. You sell yourself, then your writing. This compounds an already difficult problem for most of us. Good writers are good writers, not necessarily good salesmen. Those who spend their time selling things and have the aptitude to do so rarely also have the time or ability to write a quality book — and those who spend their time writing and have the aptitude to produce a quality book rarely have also have the time or ability to sell it.

This is a practical reality that affects almost any creative or scientific field. Those who can do can’t sell and those who can sell can’t do. But there is a deeper issue as well: a conflict of what we actually deem important with what we imagine we should deem important. What we want as authors and what we have been trained to believe we should want as Americans are largely incompatible. If we achieve only the first, we see ourselves as failures. If we achieve only the second, we see ourselves as hacks. And it is well-nigh impossible to achieve both.

It is not difficult to see why. If you’re like me and have tastes that depart even in the slightest from the mainstream, then ask yourself how many books that you really love are being published by major publishers today. Not books you’re told you’ll love, or books that you’re supposed to love, or books other people tell you they love. But books you love. For me, it’s virtually zero. The type of writing I enjoy simply isn’t published anymore. At least not by big publishers, and probably not by small presses either. It’s still being written. I’m writing it, and I’m sure plenty of other people are too. But it’s at best being self-published, and as a result is very hard to find.

The same is true of the big successes in self-publishing and is the reason neither you nor I ever will be one. The best sellers are in a small set of genres and usually involve the same perennial cast of series and authors. These authors are very good at gaming the system — i.e. at selling their books. However, they are not authors in the sense I described. They view a book as a product and nothing more. They run a business and are very good at it. For them, there is no contradiction in goals because their sole goal is financial success. There is nothing wrong with this, but it is not sufficient (or even attainable) for an author of the type I am addressing this to.

I’ll give you an example of what I mean. When I used to live in New York City, there was a famous camera store I frequented. In the same building, the next storefront was a diving shop. One day, I needed to buy some diving gear and went into that shop. I recognized some of the employees from next door, and it turned out that both stores were owned and run by the same people. The employees in either store knew everything about what they sold. If I asked an obscure question about a camera feature or model, someone knew the answer. But if I asked a subjective question, such as which camera they preferred or which BCD they found comfortable when diving, they were of little help. They could opine about which model customers preferred, and they could rave about one or the other product in a sales-pitch sort of way, but they clearly had no personal experience with the products.

I wondered at this and asked a friend who moved in similar circles about it. He explained that the product didn’t matter. It was all about understanding the market and sourcing the products at low cost. The store employees were generic highly-skilled business people. They could go into any market, learn the jargon and product specs and market layout, source the products at a good price, and then advertise and hard-sell those products very effectively. To them, it didn’t matter what they were selling. The products were widgets. The owners of those stores probably had no especial love of photography or diving but recognized those as markets they could thrive in.

Almost all self-published authors who succeed financially are of the same ilk. The books are products, and they just as happily would produce wicker baskets if that was where the money lay. Such authors have no ambition to write a high-quality literary novel. If their market research says that novels about vampire billionaires who fall for midwestern housewives are the thing, they will pump out dozens of nearly-identical ones. In this regard, such writers are a bit like the big publishers. The main differences are that (1) these self-published writers produce their own products and (2) the big publishers seem to have lost their focus these days and now employ ideological criteria rather than purely market-related ones.

The result is obvious. If I write a book of which I’m proud, it won’t get traditionally published and it won’t sell much when self-published. At a more basic level, this is a problem which affects all "producers", including artists, scientists, and musicians. To succeed in the social/financial sense, you have to spend 100% of your time relentlessly promoting yourself (and even then, the likelihood of success is small), but to produce anything of substance you have to spend all your time developing your craft and then applying it. The product-writers I described are very efficient. They are experts at what they do. After all, even in the world of marketable-schlock there is lots of competition. The winners know how to game the search engines, get in early and stay at the top, spend marketing money efficiently, and expend the minimum time necessary to produce a salable product.

The gist is that the two goals of a real writer are utterly incompatible. Writing a book we are proud of and achieving social/financial success with it are mutually exclusive for most of us. Unless you really love writing crowd-pleasing schlock or happen to be one of the handful of random literary "success" stories, it is impossible.

"But Ken," I can hear you whining, "I thought you said this would be uplifting? That my cat would still love me and my house wouldn’t burn down and I’d remember to turn off airplane mode before tweeting. How the hell is this remotely optimistic? Do you secretly run a razor-blade and cyanide business on Amazon?" Well, yes and no. Since books don’t sell, I do need some side hustles. Please visit my Amazon page for a very special offer.

Ok, fine. Here’s the inspirational bit. It isn’t that we can’t achieve both goals — it’s that we don’t have to adopt both goals. You are in control of your goals, even if your social programming reeeeally wants you to think otherwise. If your goal is to be both successful in the American sense and proud of your work, you’re going to be bitter and miserable. It’s disheartening and you’ll give up as a writer or feel resentful toward the world. But that shouldn’t surprise you. If you demand the impossible, you’ll always be disappointed. If your goal was to be a fantastic high-school teacher and also become rich from it, you’d be miserable too. It’s very hard to succeed financially in any way, let alone one which appeals to you. If I wanted to be a professional basketball player, I’d be disheartened. I’m five-foot-eight. The fault wouldn’t be with the world, it would be with me for demanding the impossible. While it’s admirable to pick difficult but attainable goals, picking wildly implausible ones is a recipe for misery. If you set out to prove the world wrong, all you’ll do is prove yourself a fool. Not because the world is right, but because there’s no point in wasting your life trying to prove anything to nine billion people who won’t notice and couldn’t care less if they did.

I’m not spouting some hippy nonsense about eschewing material possessions. You need money to survive and live comfortably. Money can buy you independence and free time. I’m not saying you don’t need money or shouldn’t pursue it. Just don’t rely on your writing for it, at least not if you want to be proud of that writing. It is perfectly fine to aim for American-style success. It’s difficult, but anything worth striving for is difficult. Nor is it unattainable, assuming your ambition isn’t too extravagant. In that’s your primary aim, do what the successful schlock-producers do and maybe you’ll succeed.

I’m also not saying you should sit in a corner munching a soggy carrot like some dejected rabbit. It is perfectly fine to write books you are proud of and hope for American-style success. I hope that my lottery ticket will win a billion dollars. There’s nothing wrong with that. It may even happen. Hope can be beneficial.

What is not fine is to expect American-style success from your writing. That is toxic. It means you’ll never reward yourself. Even if you write the greatest novel in the world, you still won’t allow yourself a sense of accomplishment. Imagine a small-town artisan who crafts beautiful furniture but demands that each piece be featured on some television show. He’ll be perpetually disappointed. No matter how great his skill and attainment, he never can give himself the slightest praise. There’s always a monkey on his shoulder telling him "So what? You’re not on television." If you’d laugh at such a person, take a good, hard look in the mirror.

To illustrate our biases, here are some scenarios. Suppose you learned that a friend …

  1. Wrote a wonderful book, got rejected by 200 agents, self-published it, and sold 3 copies.
  2. Self-published a vampire-billionaire-loves-midwestern-housewife book and sold 50,000 copies.
  3. Wrote a vapid, self-indulgent novel with elements designed to appeal to certain political sensibilities, which has been picked up by a major publisher.
  4. Self-published a book of pictures of cats with cute little taglines, which went viral and sold 100,000 copies.
  5. Wrote a passable book, though nothing worthy of note, but knew some agents and got picked up by a major publisher.

Most of us automatically would be "impressed" by (2)-(5) but view (1) as a vanity project. That is ridiculous. It’s our subconscious American training at play. Think about it. (2) may be a worthy businessman but isn’t really a writer, (4) produced a little nothing and got lucky, (5) produced tofu but knew the right people, and (3) produced what best could be termed a "vanity project" which ticked the right boxes. Of the five, only (1) produced something actually worthy of praise.

Nor are these contrived scenarios by some bitter rejectee (aka yours truly). Anyone who has contact with the publishing world knows that these are highly-realistic scenarios and that they are way more common and apropos than most of us would care to admit. So why do we view (1) this way? It’s not just our American-success programming. It’s also because of another very common scenario:

  1. Wrote complete trash, self-published it, and sold 3 copies.

(6) is what gives a bad name to self-publishing and constitutes the vast vast vast majority of self-published work. It and (2) are the reason you won’t be able to be heard above the fray or find your niche audience or sell many books.

But that isn’t as awful as it sounds. For most of history, only a privileged few even knew how to write, fewer had the means and leisure to write a book, and fewer still ever got published. Even if you wrote an incredible manuscript, without the money or connections to publish it that manuscript would end up in someone’s fireplace. So what’s different now, you may ask? Isn’t the problem the same, and only the gatekeepers and criteria have changed?

Yes and no. Yes, if you go through the gatekeepers. No, because you don’t have to. You can write a book you are proud of and self-publish it. It will be up forever as print-on-demand (and/or an ebook). You don’t have to build buzz, have a grand launch, and pray you reach critical mass before the rest of your print run gets remaindered and you end up out-of-print forever. Instead, you can put your book out there and point people to it over the years as you see fit. You can market it later when you have time or some opportunity arises. A book you are proud of will be available for anyone to purchase. Your backlist never goes away. Yes, a lot of crap gets self-published today — whereas in yesteryear only a few rich people could self-publish. But that need not bother you. Bad company does not a knave make. You’re not counting on people discovering your book by wading through all that garbage. You’re just making it available. You are the discovery mechanism. When someone asks about your book, you can point them to it. If you so choose, you can spend some money to increase the chance people will buy it. You can do this when and how you want.

And if someone at a cocktail party looks down their nose at you when you mention that you are self-published, just ask them what they’ve done lately. I wouldn’t worry too much about this happening, though. Does anyone even have cocktail parties anymore?

Incidentally, through much of the last three centuries there were no traditional publishers. Everything was self-published. But there was much less of it. Now, everyone can self-publish and everyone does. But just because a lot of other books stink doesn’t mean yours does — or that it will be viewed that way by modern, intelligent people.

In conclusion:

(1) Stop thinking of self-publication as a stigmatizing last-resort and a humiliating proof of failure. It is a tool and an opportunity. Moreover, in today’s world it is both a necessity and a reality for almost any author of substance. (2) If you write a book you are proud of, allow yourself to be proud of it. Feel successful. Decouple this sense of success from guilt or shame or anxiety about it not selling. (3) Write books you are proud of. Hope for American-style success if you wish, but do not expect it. (4) Keep writing. Write what you want to read. Be pleased that you have accomplished something.

If you complete one story you are proud of, you have accomplished more in your life than 99% of people. If you complete one book you are proud of, you have accomplished more than 99.9%. If you spend your life writing books you are proud of and allow yourself to be proud of them, you will have accomplished something almost nobody does: you will have lived a life you are proud of.

Everything everyone does is for naught, "vanity and a striving after wind," to quote Ecclesiastes. Had children? Your genes will dilute out of their progeny after a few generations. Became famous? Nobody will remember you a few years from now, and if they do it will be a mere caricature. Made a lot of money? You will is the last time you get any say how it’s spent. The best you can do is live a life you are proud of. Once you’re gone, the universe ends. It is irrelevant how many people bought your book or whether it lives on or your name is remembered.

And on that uplifting note, I once again refer you to Ken’s razor-blade and cyanide shop on Amazon. Oh, and don’t forget to leave a great review when you’re done…

Two Countries

There once were two countries, A and B, and two kinds of people, purple people and green people. Each country had both purple people and green people.

In country A, the purple people were in charge. A small group of purple people were the gatekeepers of all things, the decision makers, the managers of life.

In country B, the green people were in charge. A small group of green people were the gatekeepers of all things, the decision makers, the managers of life.

The two countries shared a large border and a free one. By ancient treaty, no visas were required and no checkpoints marred the landscape. But almost nobody ever crossed the border. A nearly insurmountable range of peaks obstructed much of the length, and strong rapids made the remainder treacherous.

Though fundamentally different in nature and culture, the majority of the purple and green people did not mind one another. Many even cherished the differences, and friendly relations were by far the norm in both countries.

The two governments were exceptions.

The purple leaders of country A portrayed green people as primitive, dangerous, and unable to restrain their impulses, creatures to be feared and controlled. The green people sought to dominate and oppress them, they warned. Only through constant vigilance and zeal could such a dire threat be averted. Whether they believed these words or simply found them politically expedient is unclear.

The green leaders of country B portrayed purple people as arrogant, irrational, and immoral, individuals of loose character and dishonest nature. Such people sought to lead good folk astray and never should be allowed influence, never should be listened to, they warned. Only through constant vigilance and zeal could such a dire threat be averted. Whether they believed these words or simply found them politically expedient is unclear.

Most green and purple people in both countries meant well, or at least did not intend ill. But a few did as a few will do, and this was exacerbated by the rhetoric of each government.

Every time a purple person in country B was attacked, the leaders of country A pointed and exclaimed “See, we are right. We must protect purple people from the inexcusable barbarity of the green people.” But they held no power in country B and compensated with an excess of zeal in their own country. Small crimes were made big, a growing range of behavior was criminalized, penalties grew, initiatives to advance purple people in the face of obvious oppression were advanced, and the public was freshly informed of the omnipresent danger posed by green people.

Every time a green person in country B was persecuted, the leaders of country B pointed and exclaimed “See, we are right. We must protect green people from the hysterical lunacy of the purple people.” But they held no power in country A and compensated with an excess of zeal in their own country. Small crimes were made big, a growing range of behavior was criminalized, penalties grew, initiatives to suppress the influence of purple people in the face of their obvious irresponsibility were advanced, and the public was freshly informed of the omnipresent evil posed by purple people.

The green people in country A cringed whenever something happened in country B. The inevitable furor surely would land on their heads. An inquisition would follow, jobs would be lost, lives would be ruined, and the slightest misstep would destroy them.

The purple people in country B cringed whenever something happened in country A. The inevitable furor surely would land on their heads. Vilification would follow, new restrictions would be imposed, rights would be lost, lives would be ruined, and the hope of improvement would grow ever more distant.

The majority of purple people in country A were not particularly swayed by their government’s propaganda, but they did not repudiate it. Most did not understand the plight of their green fellow citizens. They dismissed green complaints as hyperbolic, arguing that their government meant well and any real impact on green people was minimal. Those who believed the truth dared not speak up, and the purple leaders grew ever more powerful. Soon the green people sat hands in laps, eyes down, afraid that the slightest gesture or word could be seen as a threat by those purples who made a business of seeing threats everywhere. A few green sycophants found some small degree of success, but even they were not safe.

The majority of green people in country B were not particularly swayed by their government’s propaganda, but they did not repudiate it. Most did not understand the plight of their purple fellow citizens. They dismissed purple complaints as hysterical, arguing that their government meant well and any real impact on purple people was minimal. Those who believed the truth dared not speak up, and the green leaders grew ever more powerful. Soon the purple people sat hands in laps, eyes down, afraid that the slightest gesture or word could be seen as a sin by those greens who made a business of seeing sins everywhere. A few purple collaborators found some small degree of success, but even they were not safe.

Through the vagaries of geopolitics, some families happened to span both countries. On the rare occasions when they spoke, neither side believed the other.

The purple people in country A did not believe the tales told by their relatives in country B. These were exaggerations spread by politicians, they declared. After all, they experienced no such thing. If anything, their lives were easier than before. A few, seeing the oppression of green people in their own country (but unwilling to speak up about it) even rebuked their relatives. If anything, green people were the oppressed not the oppressors. It was one thing not to help them, but quite another to blame them.

The green people in country B did not believe the tales told by their relatives in country A. These were exaggerations spread by politicians, they declared. After all, they experienced no such thing. If anything, their lives were easier than before. A few, seeing the oppression of purple people in their own country (but unwilling to speak up about it) even rebuked their relatives. If anything, purple people were the oppressed not the oppressors. It was one thing not to help them, but quite another to blame them.

In this way, country A raced toward a dystopia for its green citizens and country B raced toward a dystopia for its purple citizens, yet nobody else recognized this.

Each government was the other’s best friend, and both were the people’s worst enemy.

This is how half the population did not realize the sky was falling, while the other half saw it happening with their own eyes.

But I apologize. I misspoke. The border has no mountains or rapids. It is not physical or legal, but one of social milieu, profession, and education. Yet it is no less real for this lack of topography. Despite the apparent freedom to do so, most people lack the wherewithal to cross the border.

The two countries are our country, today.

This is where we are, this is where we are going, and this is why you will not be believed if you say so.

A Travel-Time Metric

Especially in urban areas, two locations may be quite close geographically but difficult to travel between. I wondered whether one could create a map where, instead of physical distances, points are arranged according to some sort of travel-time between them. This would be useful for many purposes.

Unfortunately, such a mapping is mathematically impossible in general (for topological reasons). But so is a true map of the Earth, hence the need for Mercator or other projections. The first step in constructing a useful visualization is to define an appropriate Travel-Time metric function. Navigation systems frequently compute point-to-point values, but they are not bound by the need to maintain a consistent set of Travel Times between all points. That is our challenge – to construct a Travel-Time metric.

Read the Paper (PDF)

Inflation, Up Close and Personal

It often seems like the inflation figures touted by officials and economists have little connection with the real world. There are a number of reasons for this, some technical and some political. But there is a deeper problem than the means and motives for calculating any specific index. The issue is that any aggregate number is likely to deviate significantly from one’s personal experience. Each of us saves for different reasons and spends in different ways. Without taking these specific choices into account, we cannot accurately represent or protect against the inflation that we individually encounter. This paper elaborates on this idea and explains how each of us can identify the relevant components of inflation, and best hedge our savings.

Read the Paper (PDF)

A Proposal for Tax Transparency

Taxes necessarily are unpopular. They represent an economic burden and do not yield obvious benefits. Though some make a show of embracing their civic duty, few voluntarily would undertake to do so if given a choice. The criminal penalties attached to evasion and the substantial efforts at enforcement are evidence of this. Nonetheless, there is a tie between one’s sense of social responsibility and the palatability of taxes. A perception that our sacrifice benefits ourselves, our loved ones, and society as a whole can mitigate the pain it causes. Conversely, if our hard earned money vanishes into an opaque hole of possible waste and corruption, resentment is engendered.

The taxes paid by an individual represent a substantial sum to him, but a mere pittance to the government. If there is no accounting for this money, then it appears to have been squandered. This assumption is natural, as the government is known to be a notorious spendthrift. Nor does the publication of a voluminous, incomprehensible, and largely euphemistic budget lend transparency. Even if it were perfectly accurate, and every taxpayer troubled to read it, the human mind isn’t wired to accurately grasp the relationships between large numbers. Thirty thousand dollars in taxes is minuscule compared to a billion or ten billion or a hundred billion, and it makes little difference which of those quantities is involved. Therefore an effort to elicit confidence through a full disclosure of expenditures would be ill fated even if well intentioned. However it would serve to enforce accountability, and should be required in addition to any other measures employed. If nothing else, this would allow watchdog organizations to analyze government behavior and identify waste.

So how could we restore individual faith in the system of government expenditure? There is in fact a way to do so and encourage fiscal responsibility at the same time. Individuals like to know where their money went. A successful tactic of certain charities is to attach each donation to a specific child or benefit. A person feels more involved, is more likely to contribute, and is better satisfied with their contribution if it makes a tangible difference. We need to know that we aren’t wasting our money.

The pain of an involuntary contribution may be assuaged through a similar approach. It may even transform into pride. There will be individuals who remain resentful, just as there are those who do not donate to charity. And some people simply don’t like being forced to do anything. However the majority of taxpayers likely will feel better if they know precisely where their money went.

We propose that an exact disposition of each individual’s taxes be reported to him. At first glance, this may seem infeasible. Funds are drawn from pooled resources rather than attached to such specific revenue streams. However, what we suggest can be accomplished without any change in the way the government does business, and our reporting requirement would not prove onerous. The federal, state, and most local governments already meticulously account for expenses – even if they do not exhibit particular restraint in incurring them. They must do so for a variety of legal and regulatory reasons, and records generally exist even if not publicly available.

Individual tax contributions need only be linked to expenditures at the time of reporting, but this must be done consistently. To that end, expenses could be randomly matched with the taxes that paid for them. This could be done each February or March for the prior year. We simply require that each dollar of taxes collected be assigned to one and only one dollar spent and vice versa. If there is a surplus, then some taxpayers would receive an assignment of “surplus” and if there is a deficit then certain expenses will be assigned a non-tax source – such as borrowed money or a prior year’s surplus. If a taxpayer’s contribution has been marked as surplus, then his true assignment is deferred until such time as the surplus is spent (again using a lottery system for matching). If it covers a prior year’s deficit then it is matched against that year’s excess expenses. The point is that every dollar of taxpayer money eventually is matched against a real expense.

For example, one taxpayer’s report could read “10K toward the construction of 121 example plaza, New York,” or better still “3K used for the purchase of air conditioning units, 5K for ductwork, and 2K for electrical routing for work done at XXX and billed to YYY contracting on ZZZ date. Work completed on AAA date.” An individual receiving such a report would feel a sense of participation, accountability, and meaningful sacrifice.

It may seem that few people would feel pride in defraying the cost of mundane items, but such an objection is misguided. These are real expenses and represent a more comprehensible and personal form of involvement than does a tiny fraction of an abstract budget. If an expense would appear wasteful, pointless, or excessive, then it is appropriate to question it.

What of the pacifist whose money goes toward weapons or the religious individual whose taxes pay for programs that contravene his beliefs? It may seem unfair to potentially violate a taxpayer’s conscience by assigning him an unpalatable expense. But no exceptions should be made. Their money is being spent in the manner described. Whether their contribution is diluted or dedicated, they live in a society that violates their ideals and they should vote accordingly.

It is our belief that a feeling of involvement in the operation of government, along with the requisite increase in transparency, would alleviate much of the general perception of irresponsibility, excess, and unaccountability. An individual may object to his relative contribution, but the means of its use would no longer be inscrutable. This could go a long way toward restoring faith in our government.

Probabilistic Sentencing

In most real situations, we must make decisions based on partial information. We should neither allow this uncertainty to prevent action or pretend to perfect certainty in taking action. Yet in one area with a great impact on an individual’s freedom and well-being we do just that. Judges and juries are required to return an all-or-nothing verdict of guilt. They may not use their experience, intelligence, and judgment to render a level of confidence rather than a mere binary choice.

I propose adopting a sentencing mechanism based on a probabilistic assessment of guilt or innocence. This allows jurists to better express their certainty or lack thereof than does our traditional all-or-nothing verdict. The natural place to reflect such an imputed degree of guilt is in the sentencing phase. I discuss the implications of such a system as well as certain issues with implementation.

Read the Paper (PDF)

The Requirements of Effective Democracy

The current popular notion of democracy is something to the effect of “the will of the people is effected through voting.’’ Though this is a far cry from the original meaning of the word or its various incarnations through history, let’s take it as our working definition. It certainly reflects the basic approach taken in the United States. Though often confounded by the public mind with a vague cultural notion of freedom, it only conforms to this when taken together with certain other principles – such as explicit protections of individual liberties.

This aside, let us consider the components necessary for democracy. To do so, we must make some supposition regarding the ability of an individual voter to render a decision. We assume that every voting individual, regardless of aptitude, is capable of determining their purpose in voting. We say “purpose” rather than “criterion” because we refer to a moral choice, what they hope to achieve by voting. This is far more basic and reliable than any specific set of issues or criteria. A person knows their value system, even if they can not or do not have the means of accurately expressing it. The desires to improve the country, foster religious tenets, create a certain type of society, support the weak, advance one’s own interest, protect a specific right, or promote cultural development cannot easily be manipulated or instilled. While it is possible to create a sense of urgency or attach specific issues or criteria to these values, one’s purpose itself is a reflection of that individual’s view of society and their relationship with it. To meaningfully participate in the democratic process, an individual must translate this purpose into particular votes in particular elections. Note that a purpose may embody a plurality of ideals rather than any specific one (such as in the examples above).

It is the function of democracy to proportionately reflect in our governance and society the individual purposes of the citizenry. A number of components are involved, any of whose absence undermines its ability to do so. While the consequent process may retain all the trappings of a democracy, it would not truly function as one. Though it could be argued that such imperfection is natural and speaks to the shortcomings of the participants rather than a failing of the institution itself, such a claim is misguided. Regardless of cause, if the people’s will is not accurately reflected then the society does not conform to our popular notion of a democracy. Whether another system would perform better is beyond our present consideration. We simply list certain key requirements for a democracy to function as we believe it should, and allow the reader to decide the extent to which our present society satisfies them.

Note that a particular government need not directly represents the interest of every citizen, but its formation and maintenance must meaningfully do so. In some loose sense this means that (1) the effect of a citizen is independent of who that citizen is, and (2) the opinion of a majority of citizens is reflected in the actions of the government. These are neither precise requirements nor ones satisfied in practice, particularly in representative democracies. However they reflect our vague cultural concept of democracy.

The following are the major components necessary for a democracy to function as we believe it should.

Choice

Once a voter has decided upon a set of positions that reflect their purpose, they must have a means of voting accordingly. There must be sufficient choice to allow an individual to embody those positions in their vote. Furthermore, the choice must be real. Marginal candidates with no chance of winning may be useful for registering an opinion, but they do not offer true participation in the election. If there are only two major candidates then the voter’s entire purpose must be reduced to a binary decision. Only if it happens to be reflected in one of the choices at hand would their view be expressible.

If there are two major candidates and they differ only on a few issues that are of no consequence to a particular individual, then that person cannot express his purpose by voting. For example if a voter feels very strongly about issue X, and both major candidates have the same opposing position on that issue, then he cannot make his will known in that election. It may be argued that the presence of small candidates serves exactly this purpose and that if sentiment is strong enough one could prevail. This is not born out by history. In a two party system, a voter is reduced to a binary choice between two bundled sets of positions. As a more extreme example, suppose there are several major issues and the candidates agree on one of them. Even if every single person in the country holds the opposite position on that issue, their will still cannot be effected through that election. If there were no other important issues, then one or the other candidate surely would take the popular position – or a third party candidate would do so and prevail. However in the presence of other issues, this need not be the case.

Finally, there must be some reason to believe that the actions of a candidate once elected will reflect their proclaimed positions. Otherwise, it will be years before the voter can penalize them. Without such an assurance – and history certainly does not offer it – a nominal choice may not be a real one. The people then acts the part of a general who cannot move his troops, however much he may threaten or cajole them.

Information

A well-intentioned individual must have a way of locating and obtaining information whose accuracy is not in question or, if uncertain in nature, is suitably qualified. Voters must have access to accurate and sufficient information. In order to translate their purpose into a vote, an individual must be able to determine the choices available and what they actually entail. Moreover, he must be able to determine the relative importance of different issues in effecting his purpose. Fear mongering, inaccurate statistics, and general misinformation could lead him to believe that a particular issue ‘X’ is of greater import than it truly is. Instead of focusing on other issues ‘Y’ and ‘Z’ which are more germaine to his purpose, he may believe that dealing with issue ‘X’ is the most important step toward it. Similarly, if the views of candidates are obfuscated or misrepresented or the significance of events is disproportionately represented, an accurate translation of his purpose into a vote may be denied a person. Even a perfectly rational and capable voter cannot make a suitable decision in the absence of information or in the presence of inaccurate information. This said, not every vehicle should be expected to provide such information. If a person prefers to listen to a news station that reports with a particular bias, that is not the fault of the information provider – unless it does so subtly and pretends otherwise.

Aptitude

A voter must have the intelligence, critical reasoning, motivation, and general wherewithal to seek out accurate information, detect propaganda or advertising, and make an informed decision. Their perceived interest must coincide with their true interest, and their purpose be accurately represented in the choice they make. It may seem that we are advocating the disenfrachisement of a segment of the population, individuals who – while failing to meet some high standard – have valid purposes of their own which they too have the right to express. This is not the case, nor is our standard artificial. We are merely identifying a necessary ingredient, not endorsing a particular path of action. Moreover, the argument that they would be deprived of a right is a specious one. Such individuals are disenfranchised, whether or not they physically vote. They lack the ability to accurately express their purpose, and easily are misled, confused, or manipulated. At best they introduce noise, at worst their votes may systematically be exploited. A blind person may have a valid destination, but they cannot drive there.

Access

Voters must be willing and able to participate. They cannot be blocked by bureaucratic, economic, legal, or practical obstacles – especially in a way that introduces a selection bias. Their votes must be accurately tallied and their decision implemented.

Structure

Not only must the structure of the democratic process treat all voters equally, their de facto influence must be equal. Depending on the nature of the voting system, certain participants may have no real influence even if the system as a whole treats them symmetrically. A simple example would be a nation consisting of four states with blocks of 3, 3, 2, and 1 votes, where each block must vote as a unit. Regardless of the pattern of voting, citizens in the state with a single vote can never affect the outcome. If that vote is flipped, the majority always remains unchanged. This particular topic is addressed in another paper.

There certainly are many other technical and procedural requirements. However those listed above are critical components that directly determine a voter’s ability to express their will through the democratic process. In their absence, voters could be thwarted, manipulated, misled, or confused. The purpose of democracy isn’t to tally votes, but to register the will of the people. Without the choice and tools to express this will, the people can have nothing meaningful to register.

A System for Fairness in Sentencing

We often hear of cases that offend our sense of fairness – excessive sentences, minor crimes that are punished more severely than serious crimes, or two equivalent crimes that are punished very differently. Rather than attempt to solve a politically and legally intractable problem, we ask a more theoretical question: whether an individual can assign sentences in a way that seems reasonable and consistent to him.  Our system is a means of doing so.  We offer a simple algorithmic method that could be used by an individual or review board to ensure that sentences meet a common-sense standard of consistency and proportionality.

We intend to offer a less mathematical and more legally-oriented version of this article in the near future.

Read the Paper (PDF)

Why Voting Twice is a Good Thing

We should require that every bill be ratified by a second vote, one year after its original passage. It goes into effect as normal, but automatically expires if not ratified at the appropriate time.

Sometimes foolish legislation is passed in the heat of the moment or due to short term pressures. Perhaps there is an approaching election, or the media has flamed popular hysteria over some issue, or there is a demand for immediate action with no time for proper deliberation, or an important bill is held hostage to factional concerns, or legislators are falling all over one another to respond with a knee jerk reaction to some event. There are many reasons why thoughtful consideration may succumb to the influences of the moment. The consequences of such legislation can be real and long lasting. Law enforcement resources may be diverted or rights suppressed or onerous demands made on businesses. It is true that legislation may be repealed, but this requires an active effort. The same forces that induced the original legislation, though weakened by time, may threaten to damage anyone who takes the initiative to rectify it.

Here is a simple proposal that could address this problem: Every piece of legislation should be voted on a second time, one year after its original passage. This vote would serve to ratify it. By making this mandatory, the burden of attempted repeal is not placed on any individual. Rather, legislators need simply change their vote. This is less likely to create a fresh political tempest, the issue’s emotional fury long spent. When an act is passed, it goes into effect as normal. However one year from that date, it must be ratified or it will expire. Obviously this should only apply to bills for which such ratification is meaningful; there would be no point in revoting on the prior year’s budget after the money has been spent. By requiring a ratification vote, legislators are given time to breath, sit back, and consider the ramifications of a particular piece of legislation. The intervening year also may provide some flavor of its real effect. A similar approach could be used at all levels of government.

The Optics of Camera Lens Stacks (Program)

In another post, I discussed the mathematical calculation of optical parameters for a configuration of stacked lenses and camera components. As is evident from the example worked out there, the procedure is somewhat tedious. Instead, it is better to spend twice the time writing a program to do it. Fortunately I already did this and offer it to you, gentle reader, to use and criticize. I expect no less than one rabid rant about some aspect that doesn’t pedantically conform to the IEEE standard. This is working code (and has been checked over and tested to some extent). I use it. However, it is not commercial grade and was not designed with either efficiency or robustness in mind. It is quick and dirty – but graciously so.

Think of this as a mine-shaft. You enter at your own risk and by grace of the owner. And if you fall, there won’t be non-stop human interest coverage on 20 TV channels as rescue workers try to extract you. That’s because you’re not a telegenic little kid and this is a metaphor. Rather, you will end up covered in numeric slime of dubious origin. But I still won’t care.

All this said, I do appreciate constructive criticism and suggestions. Please let me know about any bugs. I don’t plan to extensively maintain this program, but I will issue fixes for significant bugs.

The program I provide is a command line unix (including MacOS) utility. It should be quite portable, as no funky libraries are involved. The program can analyze a single user-specified configuration or scan over all possible configurations from an inventory file. In the latter case, it may restrict itself to configurations accessible using the included adapters or regardless of adapter. It also may apply a filter to limit the output to “interesting” cases such as very high magnification, very wide angle, or high telephoto.

The number of configurations can be quite large, particularly when many components are available, there are no constraints, and we account for the large number of focal/zoom choices for each given stack. For this reason, it is best to constrain scans to a few components in an inventory (by commenting out the components you don’t need). For example, if one has both 10 and 25mm extension tubes then try with only one. If this looks promising, restrict yourself to the components involved and uncomment the 25mm as well.

Either through the summary option or the use of a script to select out desirable configurations, the output may be analyzed and used for practical decisions. For example, if a 10x macro lens is needed and light isn’t an issue then a 1.4X telextender followed by a 200mm zoom followed by a reversed 28mm will do the trick. It will have a high f-stop, but if those components are already owned and we don’t need a low f-stop it may be far more cost-effective option than a dedicated ultra-macro lens (there aren’t any at 10X, but a 5X one is available).

For simple viewing of the results, I recommend the use of my “tless” utility. This isn’t a shameless plug. I wrote tless for myself, and I use it extensively.

Go to Google Code Archive for Project

The Optics of Camera Lens Stacks (Analysis)

This first appeared on my tech blog. I like to play around with various configurations of camera lenses.  This partly is because I prefer to save money by using existing lenses where possible, and partly because I have a neurological condition (no doubt with some fancy name in the DSM-IV) that compels me to try to figure things out. I spent 5 years at an institute because of this problem and eventually got dumped on the street with nothing but a PhD in my pocket.  So let this be a warning: keep your problem secret and don’t seek help.

A typical DSLR (or SLR) owner has a variety of lenses.  Stacking these in various ways can achieve interesting effects, simulate expensive lenses (which may internally be similar to such a stack), or obtain very high magnifications.  Using 3 or 4 lenses, a telextender, a closeup lens, and maybe some extension rings (along with whatever inexpensive adapter rings are needed), a wide variety of combinations can be constructed.  In another entry, I’ll offer a companion piece of freeware that enumerates the possible configurations and computes their optical properties.

In the present piece, I examine the theory behind the determination of those properties for any particular setup.  Given a set of components (possibly reversed) and some readily available information about them and the camera, we deduce appropriate optical matrices, construct an effective matrix for the system, and extract the overall optical properties – such as focal length, nearest object distance, and maximum magnification.  We account for focal play and zoom ranges as needed.

The exposition is self-contained, although this is not a course on optics and I simply list basic results.  Rather, I focus on the application of matrix optics to real camera lenses.  I also include a detailed example of a calculation.

As far as I am aware, this is the only treatment of its kind.  Many articles discuss matrix methods or the practical aspects of reversing lenses for macro photography.  However, I have yet to come across a discussion of how to deduce the matrix for a camera lens and vice-versa.

After reading the piece, you may wonder whether it is worth the effort to perform such a calculation.  Wouldn’t it be easier to simply try the configurations?  To modify the common adage, a month on the computer can often save an hour in the lab.  The short answer is yes and no.  No I’m not an economist, why do you ask?

If you have a specific configuration in mind, then trying it is easier.  However, if you have a set of components and want to determine which of the hundreds of possible configurations are candidates for a given use (just because the calculation works, doesn’t mean the optical quality is decent), or which additional components one could buy to make best use of each dollar, or which adapter rings are needed, or what end of the focal ranges to use, then the calculation is helpful.  Do I recommend doing it by hand?  No.  I even used a perl script to generate the results for the example.  As mentioned, a freeware program to accomplish this task in a more robust manner will be forthcoming.  Think of the present piece as the technical manual for it.

Tless Table Viewer

Over the years, I’ve found delimited text files to be an easy way to store or output small amounts of data. Unlike SQL databases, XML, or a variety of other formats, they are human readable. Many of my applications and scripts generate these text tables, as do countless other applications. Often there is a header row and a couple of columns that would best be kept fixed while scrolling. One way to view such files is to pull them into a spreadsheet, parse them, and then split the screen. This is slow and clumsy, and updates are inconvenient to process. Instead, I wanted an application like the unix utility ‘less’ but with an awareness of table columns. The main requirements were that it be lightweight (i.e. keep minimal content in memory and start quickly), parse a variety of text file formats, provide easy synchronized scrolling of columns and rows, and allow horizontal motion by columns. Strangely, no such utility existed. Even Emacs and vi don’t provide an easy solution. So I wrote my own unix terminal application. I tried to keep the key mappings as true to “less” (and hence vi) as possible. The code is based on ncurses and fairly portable. The project is hosted on Google Code and is open source.

Go to Google Code Archive for this Project

Influence in Voting

Have you ever wondered what really is meant by a “deciding vote” on the Supreme Court or a “swing State” in a presidential election? These terms are bandied about by the media, but their meaning isn’t obvious. After all, every vote is equal, isn’t it? I decided to explore this question back in 2004 during the election year media bombardment. What started as a simple inquiry quickly grew into a substantial project. The result was an article on the subject, which I feel codifies the desired understanding. The paper contains a rigorous mathematical framework for block voting systems (such as the electoral college), a definition of “influence”, and a statistical analysis of the majority of elections through 2004. The work is original, but not necessarily novel. Most if not all has probably been accomplished in the existing literature on voting theory. This said, it may be of interest to a technical individual interested in the subject. It is self-contained, complete, and written from the standpoint of a non-expert in the field. For those who wish to go further, my definition of “influence” is related to the concept of “voting power” in the literature (though I am unaware of any analogue to my statistical definition).

Ye Olde Physics Papers

Once upon a time there was a physicist. He was productive and happy and dwelt in a land filled with improbably proportioned and overly cheerful forest creatures. Then a great famine of funding occurred and the dark forces of string theory took power and he was cast forth into the wild as a heretic. There he fought megalomaniacs and bureaucracies and had many grand adventures that appear strangely inconsistent on close inspection. The hero that emerged has the substance of legend.

But back to me. I experienced a similar situation as a young physicist, but in modern English and without the hero bit.   However, once upon a time I DID write physics papers. This is their story…

My research was in an area called Renormalization Group theory (for those familiar with the subject, that’s the “momentum-space” RG of Quantum Field Theory, rather than the position-space version commonly employed in Statistical Mechanics – although the two are closely related).

In simple terms, one could describe the state of modern physics (then and now) as centering around two major theories: the Standard Model of particle physics, which describes the microscopic behavior of the electromagnetic, weak, and strong forces, and General Relativity, which describes the large scale behavior of gravity. These theories explain all applicable evidence to date, and no prediction they make has been excluded by observation (though almost all our effort has focused on a particular class of experiment, so this may not be as impressive as it seems). In this sense, they are complete and correct. However, they are unsatisfactory.  

Their shortcomings are embodied in two of the major problems of modern physics (then and now): the origin of the Standard Model and a unification of Quantum Field Theory with General Relativity (Quantum Field Theory itself is the unification of Quantum Mechanics with Special Relativity). My focus was on the former problem.  

The Standard Model is not philosophically satisfying. Besides the Higgs particle, which is a critical component but has yet to be discovered, there is a deeper issue. The Standard Model involves a large number of empirical inputs (about 21, depending on how you count them), such as the masses of leptons and quarks, various coupling constants, and so on. It also involves a specific non-trivial set of gauge groups, and doesn’t really unify the strong force and electro-weak force (which is a proper unification of the electromagnetic and weak forces). Instead, they’re just kind of slapped together. In this sense, it’s too arbitrary. We’d like to derive the entire thing from simple assumptions about the universe and maybe one energy scale.

There have been various attempts at this. Our approach was to look for a “fixed point”. By studying which theories are consistent as we include higher and higher energies, we hoped to narrow the field from really really big to less really really big – where “less really really big” is 1. My thesis and papers were a first shot at this, using a simple version of Quantum Field Theory called scalar field theory (which coincidentally is useful in it’s own right, as the Higgs particle is a scalar particle). We came up with some interesting results before the aforementioned cataclysms led to my exile into finance.

Unfortunately, because of the vagaries of copyright law I’m not allowed to include my actual papers. But I can include links. The papers were published in Physical Review D and Physical Review Letters. When you choose to build upon this Earth Shattering work, be sure to cite those. They also appeared on the LANL preprint server, which provides free access to their contents. Finally, my thesis itself is available. Anyone can view it, but only MIT community members can download or print it. Naturally, signed editions are worth well into 12 figures. So print and sign one right away.

First Paper on LANL (free content)
Second Paper on LANL (free content)
Third Paper on LANL (free content)
First Paper on Spires
Second Paper on Spires
Third Paper on Spires
Link to my Thesis at MIT