We see them fleetingly. Men dressed in white sheets, paunchy figures in survival gear, throwing their Hitler salutes, almost unreal. We laugh or shake our heads, turn the page, and move on to less depressing or ridiculous news.

Like you, so I once thought. Sniggering at the TV, safe and comfortable, the world reduced to bite-sized headlines. After all, what have skinheads, hooligans and Nazi fanatics to do with the rest of us?

But what if – just if – these people represented something larger? A twisted mirror to al-Qaeda, perhaps. Leading ordinary lives in and among us, with the same problems and fears. Not so different after all.

My book Homeland tells the story of my journey into their world. A glimpse into a place most of us will not, or choose not, to see. Those travels also led to my collaboration with the BBC, as creative producer for the new BBC1 drama England Expects. Written by the accomplished Irish screenwriter, Frank Deasy (The Captives, Looking After Jo Jo, Real Men, Prozac Nation), this is the first major TV drama to tackle the far right in over 20 years. It was back in 1982 that a young Tim Roth made a fearsome debut playing the eloquent skinhead Trevor, in Alan Clarke’s film Made in Britain. Little has been attempted since.

In the character of East End security guard Ray Knight (Steven Mackintosh), Deasy creates a frighteningly believable extremist: a man who loves his daughter, has a job, tries hard (but fails) to understand the world around him, and slowly descends back to his far-right roots as his life collapses.

Through Ray’s eyes we meet someone not immediately dissimilar to our friends and neighbours. Of course, all is not well with Ray, there are issues linked to his past, and a crippling desire for control, which cause him to lash out. He also is searching for a lost identity, a common theme among many of the extremists I met. Like those supporting the far right today, Ray has to pin blame on the ‘Other’: in his case, the Bangladeshi Muslim community living around him in east London. For some, it is simply Muslims per se. Or Asians, immigrants, benefit tourists and terrorists. The list could go on. I have seen its like repeated by white voters in the north west of England, complaining about non-existent asylum seekers, all the way to radicals in The Lebanon, who talk of a Zionist conspiracy and want to see Israel disappear from the face of the Earth.

Ray’s journey is a ‘what if?’, posing an important question about the nature and direction of the far right today, how it plays on our fears and actively stokes up tensions. Although very much a drama, England Expects has real research behind it. Frank Deasy and I met members of extremist parties and organisations across Britain: on the campaign trail, in pubs, at their homes, and at their meetings. We have seen how groups leaflet and target problem areas: it is well known that far-right gangs were instrumental in setting off race riots in Oldham in 2001, part of a summer of discontent that saw Burnley and Bradford also go up in flames as young Asians vented their fury on police. Both of us have also been into and among the Bangladeshis of the East End – where there is an identity crisis among the young – and talked with many Muslims.

It is a strange feeling to know the drama is finally appearing. It was back in August 2000 that a meeting took place between myself and veteran BBC drama producer Ruth Caleb. At that time Ruth simply had an idea to make something on the far right. Like my own journeys, neither of us quite knew just how prescient it might be. Now many months, and years, later the result is out. For most of that time, few even inside the BBC knew what we were preparing. The project was shrouded in secrecy, going by the name Rays Daze. Extra security was hired for parts of the filming. But for me it was simply the culmination of a series of encounters that had started years before.

"Stitch us up, and we'll fuck you over badly," the voice had growled. It belonged to a fat, leering face, the mouth set in a small, intimidating 'o'. Sour breath had wafted over me. Paul David 'Charlie' Sargent was a leader not so much by charisma as by force and fear. Around him a clamour of guttural "yeahs" supported his words.

The time was autumn 1996. Unknown to my friends or family, I had arranged a meeting with the leadership of Combat 18, Britain's most notorious neo-nazi gang. The '1' and '8' in their name stood for the initials 'A' and 'H' in the alphabet, those of Adolf Hitler. I had contacted the anti-fascist magazine Searchlight, looking for information on football hooligans. I was hoping to write a book on our growing subcultures, wanting to go in-depth and behind the usual headlines. The young man who called back suggested I write a story about the gang and its desire for an "Aryan homeland" in Essex. Laughable now. But ethnic homelands and racial tensions – the idea of separation – were themes I encountered time and again on my travels. Not just in the wilds of Alabama, the usual fare stoked up by other writers and documentary makers poking fun at cranky Americans, but alive in the hatred spilling out in Kosovo, Bosnia, Chechnya, Rwanda and a thousand other conflicts.

I was to be the first outsider to meet Combat 18, winning the trust of the two Sargent brothers who led the enterprise. Sitting in a pub surrounded by drinking commuters, it was a strange – and threatening – entrance into the world of right-wing extremism. Terrified would be too simple a way to describe my feelings. Fascinated and terrified might be a better way to put it. Yet the gang clearly wanted to talk, to share their thoughts. In end, I was amazed how easily they opened up. Few, it seemed, had bothered to approach them in the way I’d done.

With their links to Loyalist paramilitaries, hooligan ‘firms’, continental neo-nazis and a burgeoning illegal music business, it couldn’t last of course. Charlie Sargent fell out with his lieutenant, a man nicknamed ‘The Beast’. The feud led to a murder – in and around my visits to the brothers – for which Sargent is now serving a life sentence.

In the end, Combat 18 became my introduction to something much larger. To Nick Griffin, for example, the leader of the British National Party (BNP) and former vice-chairman of the National Front. It was my ‘friendship’ with Griffin – who clearly saw me as a liberal journalist he could manipulate – that a doorway opened into a vast network.

This was a world which seemed to mirror our own. But it was also a reflection of al-Qaeda. As fundamentalism rose in the East, so our own zealots grew in the West. The network stretched from the most violent and deranged hooligans and bombers, through to suave politicians and presidential candidates. With rising xenophobia, the irony remained that as some blamed immigrants for taking their jobs (and somehow at the same time all our dole money), those jobs might well be relocating overseas anyway.

My travels would bring me to places where the boundaries with my own beliefs sometimes blurred. Where 'respectable' folk could be racists; where I swapped gossip about beer prices and TV programmes with men who could squirt acid into the face of an Asian woman, then laugh about it. Most of the time, though, I discovered that hate wears a different face altogether.

In this world, I found extremists of all shades mixing: fascists with vegans, animal liberation extremists with anti-abortionists, and a mêlée of small far-left, anarchist, far-right and environmental groups on the fringes of the anti-capitalist crusades. During my six-year journey, millions of my fellow Europeans – people who loved their children, worked nine to five, and thought of themselves as respectable citizens – voted for neo-nazi and ultra-nationalist parties.

Which was my world and which theirs? I often couldn't tell. Perhaps it was fear that drove them. I sensed it all around me, a cancer that seeped into our lives, allowing us to do terrible things. In the wings, lurking behind the smiles and smooth talk of the racist politicians, I discovered the pale faces of the boy-next-door killers, barely men, burning with frustration, needing belief, prepared to act. Loners and zealots like London nailbomber David Copeland, who so nearly killed friends of mine, or Oklahoma bomber Timothy McVeigh.

Thanks to my connections with Nick Griffin, I even lived with the BNP's man in America. Mark Cotterill, a Loyalist supporter and former NF member, was working for ex-Republican Pat Buchanan’s presidential campaign. I met Buchanan, then accompanied Cotterill down to the annual conference of the Council of Conservative Citizens, a kind of white collar KKK. I met many people, too, linked to the issue of Holocaust denial, including a former White House press officer and a Republican lobbyist. Cotterill is now back in the UK, backing a rival far-right party against the BNP.

My unmasking – if you can call it that (it was revealed that I had had help from the Right’s mortal enemy, Searchlight) – took place in a hotel in North Carolina. Members of the world's most notorious neo-nazi cult, the National Alliance, were extremely hostile to my presence and said they knew all about me (they didn’t). Before his death two years ago, the Alliance was run by neo-nazi ideologue, William Pierce. His book, The Turner Diaries, about a worldwide racist uprising, was found in the possession of the Oklahoma bomber and in the flat of David Copeland. My encounter with Pierce's then-number two man was extremely unpleasant, but thankfully short-lived.

Deep in the mountains of Arkansas, I visited a fundamentalist, racist Christian ministry. David Copeland had once surfed its site, something he shared with his police interrogators as he sought to justify his actions on racial and religious grounds. I was called undercover to Beirut for a conference of Holocaust deniers and Islamic fundamentalists, united in mutual anti-Semitism. Later in my voyage, I crossed paths with Europe's smiling ultra-nationalist politicians. And at the end of this strange odyssey I gained access inside the "liberated zones" rising in the former East Germany, where a founding member of the Baader-Meinhof gang was helping to lead a neo-nazi political party. Someone our very own Nick Griffin had encountered at a gathering of international extremists.

Everywhere I looked, and turned, hate seemed to surround me. Many of the people I met seemed to be searching for something: belonging, power, a second of meaning or glory in their lonely and frightening world. Of course, there are many real reasons and causes – the failures of the Left, mass voter apathy, the breakdown of traditional community, plus the rise of single issue politics and a harking back to ‘mythical’ better times – that suggest why people are voting for the Right. Still, I could not help but be struck by the sheer number of angry, intense and deeply repressed individuals flocking to this scene.

Are they dangerous? Well, as the drama England Expects hopefully shows, these groups are a warning sign; of pressures building. And you meet very real and dangerous people within them. The politicians may have ‘suited up’, like Keith Barron’s character Larry, but you still find the lone wolf accepted within their ranks. People like London nailbomber David Copeland, for example, who belonged both to the BNP and then the National Socialist Movement (a tiny group spawned by Combat 18 and run by a former monk). So-called 'lone wolves' are undoubtedly attracted to, and influenced by, fundamentalist organisations. These add to the momentum that helps tip potential killers over the edge.

Yet every right-wing leader I met tried to deny they bore any responsibility for some nut actually taking them at their word – i.e. starting a race war. You don't see Copeland, or Buford Furrow (a white supremacist who shot Jewish children in a kindergarten in LA), Maxime Brunerie (the French neo-nazi who tried to assassinate French president Jacques Chirac on Bastille Day 2002) or Thomas Nakaba (a Danish member of C18, a bomber whom I followed in my travels) in mainstream political parties. You find them on the fringe.

What I discovered – or had affirmed – during the research for my book, Homeland, was that I was witnessing the distorted face of belief. There were people, whether neo-nazi skins or leaders of xenophobic political movements, who wanted to contain our often complex and confusing world with a black-and-white straitjacket.

Those people will probably dismiss me, regard me as a traitor, or try to shoehorn me into their two-dimensional world view. A German neo-nazi has already posted a message onto the Net which says someone should “knife this c***”. A Canadian extremist on the Stormfront ‘white pride’ website reckons: “Nick Ryan is a man with a world of hate for the White race”.

Hate does come easily. I saw its victims travelling in convoy down through Slovenia, Croatia and into Bosnia, in the lines of refugees crawling over the border from Kosovo into Albania, when I met families whose sons and daughters had had their throats cut by Islamic fundamentalists (and government death squads) in Algeria. Yet the people I met in the realms of ultra-nationalism would say that they "loved" their race. If Hitler came to power again, many had privately joked to me, most of their comrades today would be the first up against the wall.

By the end of my journey, BNP leader Nick Griffin was thanking the tabloids for doing his job for him. We seemed to be drowning in a tide of hysteria and isolationism. I had seen how safe lives in leafy suburbs were built on something darker.

And since completing Homeland, the Right has continued its rise, its servants more capable now than mere thugs and hooligans. The line between extremism and the mainstream – us – is increasingly blurred. Fear and xenophobia are suckled by eternal war. Identity fractures, Saddam is captured and patriotism has become the elixir of choice.

Racism has reached beyond the neo-nazi fringe, far beyond the BNP’s 18 council seats. Anti-Semitic incidents are rising across Europe, the attacks often perpetrated by disaffected young Muslims. Sometimes Jews face harassment from both neo-nazis and militant Islam. In Antwerp, after finishing my book, I witnessed for myself how members of the Orthodox Jewish community had turned towards the far-right Vlaams Blok party (which has a third of all votes in the city) for protection. The Blok took its largest ever vote that same year. I talked with Muslims in Belgium and on the streets of the East End in London. These are strange times, forging stranger alliances.

The coming decades will be a time of identity politics, identity beliefs. Let us take stock now. As George Orwell once said: ‘If you want a picture of the future, imagine a boot stamping on a human face – for ever.’ Perhaps Orwell’s predictions are nearer than we think.

This article was commissioned for The Scotsman©2004

Newsletter To keep in touch with new projects, columns and other regular developments, join my newsletter.