FINISHED TRANSCRIPT
EIGHTH INTERNET GOVERNANCE FORUM
BALI, INDONESIA
BUILDING BRIDGES - ENHANCINGMULTI-STAKEHOLDER COOPERATION FOR GROWTH AND SUSTAINABLE DEVELOPMENT
OCTOBER 23, 2013
WS112
INTERNET EXCHANGE POINTS AND THE DOMESTIC INTERNET ECONOMY
* * *
This text is being provided in a rough draft format. Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
* * *
>> BILL WOODCOCK: We'll give it a couple more minutes and then get started. Let me look in the hallway and see how people are coming along. All right. I think we'll get started here.
The purpose of the session today is to talk about Internet Exchange Points and the Domestic Internet Economy, the beneficial effect they have on domestic economies. We have five people up on the panel today. Myself, I'm Bill Woodcock with Packet Clearing House, and helping with the door here is See Kiat Yeo, who is with the Singapore Internet exchange, SGIX.
To his right, your left, is Sam Paltridge, the Communications Economist at the Organisation for Economic Cooperation and Development, OECD, in Paris. To his right, your left, is Garin Ganis, who is with the Internet Users Association of Indonesia but was also operating one of the Internet service providers that was an original participant in the Jakarta Internet Exchange, one of the early, really large Internet exchanges in Asia in terms of number of participants.
To his right, your left, is David Satola, lead counsel at the World Bank and he takes a interest in exchange points and their economic development effects.
So we'll get started. Sam has just a few slides. He's up first.
>> SAM PALTRIDGE: Good morning, everyone. Is that better? Good morning, everyone. So I thought we'd play a game to start with. I don't know how many of you come from the UK but there's a television show where it's a missing word round and I put out news stories and then people try to guess the word missing. And here I've put a headline and actually made it easy, put multiple choices, and the headline is about, It's time to break the U.S. grip on the Internet says... guess who? I put a few choices there. I put OECD, ITU, Brazil, it could be another choice, it could be Iran, I don't know who you might guess. Let me give you the answer.
Bill, next slide. Actually it was OECD! Back in 1998.
Now, we don't get to choose the headlines people put to the work we do. You know from your own experience with the media that they don't necessarily choose a headline you'd choose yourself, I would not have necessarily chosen this headline but what this story was about -- and the story was actually quite accurate -- and what we were saying was we need to expand Internet infrastructure around the world, outside the U.S., and this was in 1998 and this was after a couple years of work where we had looked at the benefits of Internet exchange points and we were starting to get inquiries at the OECD from especially traditional telco operators that didn't understand the business model of the Internet or countries saying how can we join the Internet and saying but all our traffic goes to the U.S. and this is expensive.
What is the solution?
We said, okay, we looked to the Technical Community and said what's best practice here? They said, well create some Internet exchange points. We quickly learned about this and our purpose in doing this report was to say to initially OECD governments but anyone else who cared to read the report and I know Kenya did quite early on was to say exchange points are just something that's essential. We also wanted to expand things like route service. At that stage there were only 13 route servers and we said they should be expanded around the world, too, get this infrastructure out there, so that's not generating traffic and creates a more resilient Internet and today there are more than 1,000 Internet route servers.
Next slide, Bill.
One more game. One more "guess who this was." This is from an article a few weeks ago. Really this made a similar point about the debate going on around Brazil and some of the things Brazilian government is doing and I won't keep you in suspense, next slide. It was our moderator Bill Woodcock who wrote that. I don't think Bill necessarily would have chosen this headline. In fact, he told me before that he vetoed quite a number of the headlines. But the media does what it does but if you read Bill's Article, it was basically saying the same thing, consistently, that he's been saying for probably 20 years. And that is Internet exchange points are just best and good practice and what Brazil is doing is creating more international infrastructure in and out of the country, creating exchange points within the country so they don't have to import data.
Cost doesn't fall on the Internet community there. So next slide.
So essentially the main message from my five minutes or ten minutes here is that Internet exchange points are essential. Brazil has gone -- Bill, I think the numbers are from your article -- so they have gone from one Internet exchange point in 2004 to 23 today. Keeping the traffic cycle, giving Internet users better performance, lowering costs and it's just good practice. It's what people should do. It's not about vulcanizing the Internet or other things you read in the media. It is simply what the whole stakeholder group around the Internet ecosystem should be encouraging.
But even in OECD countries, we still have one OECD country, I think I'm still right in saying it doesn't have an Internet exchange point. This has implications for their economy, I think, and I know they are going to try to address this but let me show you a quick couple examples why this is important in broader economic terms. This is a map that we got a company called Pingdom to do because we were interested in where data was being hosted around the world and so what we said to them was let's take the country code names, let's take the top million in Alexa, and say okay, where are they hosted? And if the blue is darker, you're hosting more of your CC content, most popular content. If it's redder someone else is hosting that and that's expensive for you because every time someone a local person goes to access that content they have to bring it all the way from where it's hosted.
Now, I don't know the factors for all these countries. Many people in the panel will talk about this and they are more familiar with these countries. I know Yemen is probably not a leader in the Internet so maybe they only have one site in the top million and it's hosted locally but I do know a bit about the OECD countries. And there are three that don't have more than 50% of their content hosted locally. Those three countries are Mexico, Canada, and Greece.
So let me talk about Mexico first.
Mexico is the only OECD country that does not have an Internet exchange point. All content basically gets -- goes across to the states or whatever it goes but not hosted locally. Canada. Well, only marginally but just over 50% of their content with SCC is hosted by another country and of course it's the U.S. And again, I think I read a report that Packet Clearing House did where they advised the Canadian government on how to better structure their Internet exchange points so that more people would feel comfortable using the local data, lower cost, better efficiencies and so forth.
The other country -- and I really don't know the answer to this -- is Greece. So two countries host more content under the Greek CC than the Greeks. They are the U.S. And Germany. Now, this is very good exchange point. In Germany. So possibly that is a factor, Deutsch Telecom owns the major telco in Greece, there could be other factors but basically I think this is just one way you can argue if you create a local exchange point the content will stay local if it can because it wants to.
Final example and last slide, the country that I think has been the most successful in exploiting communication efficiencies over the last decade in terms of generating industry, jobs, economic growth, is in many ways India, directly related to communications and it's all to do with outsourcing and Call Centers and so forth. And if you reduce the costs of communication traffic, you have a shot at building services industry around communications infrastructure. This I not directly an Internet, but what I'm showing is calls from the United States to India and calls to Africa in terms of minutes, so you can see if you go back a decade, the U.S. made less calls to India than they did to Africa. A whole continent. That's the number of calls in minutes to Africa.
What's happened with India? It's gone off the chart! It's amazing. What's happened to the cost per minute of delivering that traffic? Well for India it's gone down like that per minute, and for Africa, it's actually increased. There's a real challenge for all countries. If you have a more efficient communication infrastructure and more efficient market you will create economic opportunities and Internet exchange points are a big part of that.
Bill, that's all I wanted to say.
>> BILL WOODCOCK: Thanks, we're trying to keep all this relatively brief so we can have fairly open discussion at the end. I have just one slide and I'll sort of speak around this.
Again, we're trying to focus on the economic part of exchange points and exchange points are analogous to factories. They are the point of production of the value of the Internet, value of the Internet being the Internet bandwidth service. So exchange points are the point of interconnection between service providers and it's that point of interconnection that creates the value customers are buying. The work that Internet service providers do is they haul traffic from its origin, which economically is the Internet exchange point, to the customer. So if we were in ideal circumstances the customer would be at the Internet exchange point and they could just partake of the value directly right there. But users don't want to sit around in the middle of a data center? Where, they want to be out working in their office or driving in their car so the job of the service provider is to move the bandwidth from the exchange point to where the user wants to consume it.
This is analogous to almost any sort of consumer good or produce or whatever. If you want a banana, you don't want to eat it in a plantation; you want to eat the banana in your house. Someone has to move that banana from the plantation to where you want to consume it.
So having a local source of Internet bandwidth means you are not importing Internet bandwidth as much. So if you look country-by-country, Germany is the country that produces the most Internet bandwidth. But Germany also has a fairly large population. Very industrialized country, lot of middle class people who get on the Web at night and read the news, play games, use the Internet a lot at work.
Germany also consumes a lot of Internet bandwidth.
Second largest producer of bandwidth is the Netherlands. They have a tiny population compared to Germany so even though each person in Netherlands is using even more bandwidth as the people in Germany they don't add up to it. So Netherlands is the world's largest exporter of bandwidth. Being a net exporter means there is inbound revenue to the country for the services that are being sold. The Internet bandwidth.
By contrast, if you look at countries that don't have Internet exchange points, Garin later will give an example of Afghanistan, prices are very, very high and the suppliers are outside the country which means we have a net import of Internet bandwidth and net export of capital. That is really not a good thing for a country where capital is already a very scarce thing. So with that sort of basic overview of exchange points as source of production, I'll run through a few of the other benefits of having them local.
Speed times distance equals cost in moving bandwidth. So you can go a long distance at a low speed for a sort of medium cost or you can go a short distance at a high speed for a medium cost or long distance at high speed for very high cost or you can go short distance at low speed for very low cost. Speed times distance equals cost.
If you have an Internet exchange point that's very near you that means you can either go very, very fast or you can go very, very inexpensively. Or both. So the shorter paths are also less expensive paths. Shorter paths are also faster because everything is constrained by the speed of light. So if you want your Web pages to load more quickly you want them to be hosted on the other side of an Internet exchange point that is near you. So that the speed of light is not causing a big delay in every transaction and those transactions add up to get every page.
So shorter, faster paths mean things are more reliable, fewer components in the network between you and the thing you are trying to reach. Each is a potential source of unreliability, each component is a potential point of failure so the fewer points of failure there are between you and whatever you are trying to reach the more of the time you can reach it successfully.
With the economic benefits the main thing is improving balance of trade. You can think of Internet as any other commodity. You are trying to export more of it than you are importing. You are trying to export more than you consume. You are trying to sell it at a higher price than you are buying it at. No different than anything else.
And obviously it's a layered thing, right, the actual transport which is what we talk about with Internet exchange points is a building block but on top of that, if you have inexpensive access to the Internet that facilitates many other kinds of businesses being layered on top.
So a test I often like to use for regulators who are trying to think about whether their regulatory rules and schemes are fair or will encourage economic development is to ask: How would this affect a high school student? How would this affect a 17-year-old? You think of the big successful content businesses on the Internet, many of them were started by people who were still students. So if you want those kinds of businesses to be started in your country you need to think about how your rules and economic environment affect people who are students, whether they have equal access to opportunities as for instance your incumbent telco has. If it is easier for your incumbent telco to do something than for a 17-year-old to do something, you won't have the kind of innovation you'd like.
So over this building block of cheap Internet access we can have additional building blocks of content development and content distribution and new protocols being written and new games being developed and new ways of sharing music and film and so forth.
That takes us to the social benefits. If it's easy and cheap to move content back and forth between people who are local to each other, and it is relatively expensive and slow to get content from somewhere else, this means that there's a locality quality to the content. That means content that is near you is cheaper and faster and more reliable and better than content you have to get from somewhere halfway around the world. That means that unlike compact discs in the hold of a freighter, Internet content actually incentivizes this local culture.
I talk with officials in the Jamaican government and for Jamaica, one of their biggest exports is music. You think of the United States as being this huge exporter of film and music and other kinds of popular culture. But there are other countries that on a sort of relative to the size of their economy do it much more. Jamaica is one of those. Iceland has a huge film industry that exports a lot of film. So for those countries it's really important to find ways to get bandwidth out so that they can stream these kinds of content that their people are creating at a cost that is not prohibitive. If they don't have exchange points and don't have networks building out to them, then they have to take that content and host it and serve it out of another country and that country gets all the economic benefit of the production and delivery, just like if Jamaica didn't have record producers and distribution labels in Jamaica, the money would all be made by U.S. and British companies that would be doing that function.
So then we go into a very current and topical thing, privacy benefits.
If you have an Internet exchange point locally and you want to talk to someone nearby and that traffic goes through your local exchange point, then only your local law enforcement will have access to that information, whatever it is you are saying. If, on the other hand, you are using exchange points in the U.S. or Germany or the UK or Japan, then those countries and every intervening country, every country between you and there, will have access to whatever it is that you were saying to whomever it is you were talking to.
If we go back historically one of the first countries to realize how serious an issue that was was Sweden. So in 2000, the Swedish government had PCH go in and look at the situation there which this was the end of the dot com era when everybody had more money than time and attention and the one exchange point in Stockholm had gotten kind of screwed up. There were technical problems with it and so all ISPs in Sweden were saying well we can't afford the time and attention to fix this. We'll just throw a little bit more money at the problem and ship all traffic down to London and back again.
So if you were in Sweden and doing business with someone across the street or even in the cubicle next to you it might well be that traffic was going all the way down to London and back again.
This was right after the Clinton administration had been found to be doing data collection with the other 5-I countries and sharing it with private industry in the U.S. so they were using national resources to spy on competitors of U.S. companies and give the information about those competitors to the U.S. companies to make them more competitive. This is something we know that China does routinely as well. This is not to single out the U.S. as being particularly bad this way. This is just to make the point that this kind of thing happens and it is in the national interest to keep the traffic local if for no other reason than for the privacy of the companies operating and the individuals in that country.
That's actually all I had prepared so I'll hand it over to David.
>> DAVID SATOLA: Good morning, everyone.
I'd like to thank the organizers of this workshop and also the government of Indonesia for hosting us here today.
My name is David Satola, I'm from the World Bank, in the legal department. I cover globally our investments and our financing of telecommunications and ICT development projects around the world.
I want to step back and talk at a somewhat higher level before getting into some of the particular issues on IXPs. We look at, we're financing right now Internet investment and initiatives in about 50 countries currently. Most of those are in the area of infrastructure investment, most of that is in broadband capacity, whether submarine cable connectivity or domestic fiber connectivity. Along with that we look, try and have a holistic approach to our projects in countries.
Looking at the enabling environment and that's the institutional environment, legal and regulatory environment, also the human capacity part of it.
So if you look at this problem of IXPs from a kind of ecosystem perspective, I get to one of the issues that Bill raised and one of the questions that Sam implicitly raised: Why are half of the countries in the world without IXPs? For all the benefits that they can bring and for all the good that they can bring to a country both in terms of connectivity, economic benefit, et cetera, what is the problem? Why are we in this day and age, in 2013, facing a situation where so few countries relatively speaking have IXPs?
Especially when you consider as Bill pointed out, the economic benefits of having good Internet for a country. We did a study a few years ago, looking, and this is sort of the headline economic news that came out of it, for every 10% of broadband penetration, you get about 1.38% economic growth cumulatively. So for 20% broadband increased penetration you get just over almost 3% economic growth. For most countries, that is headline news that cannot be ignored.
In that context, if a country is taking a policy initiative to build out Internet infrastructure and bring connectivity into the country, where do the IXPs fit into this? We didn't yet talk about the cost of actually establishing an IXP in a country and our experience from the few that we have been involved with is that relatively speaking the cost is pretty low, can be low. We're not talking about huge investment in machinery or equipment. We're not talking about huge investment in personnel or institutions, so to me there's a question there about why more of this isn't occurring.
I think part of the answer is a self-reinforcing conundrum. There are problems of scale. There are problems of capacity, both broadband and human, and there are problems of distance. These are all issues that have been put on the table already; yet, at the same time, this one little component of the Internet ecosystem, IXP, can overcome those very problems that would prevent Internet growth from happening.
It can overcome the problem of scale, overcome the problem of capacity and the problem of distance. So with low cost, then I would, I mean, if establishing an IXP, physical infrastructure, is relatively low cost I'd look at some other issues around it, which are what is the business model for making an IXP work? And then accordingly what are sort of the governance principles that go around that in terms of how the -- how it's owned, operated, who gets to participate in it, at what cost, and what are some of the barriers to getting those IXPs established.
I was in the Solomon Islands last week working on a cable project there, currently the Solomons pulls down all capacity from satellite. It's very expensive and it's prone to disturbance from weather, and the Solomon Islands is known to have a good amount of weather so there's always an issue with the satellite capacity being brought down. In that context, what could a country like the Solomon Islands and this probably affects countries throughout the Pacific, do in terms of establishing an IXP to overcome some of those issues. We are dealing in about eight or ten countries in the Pacific, both South Pacific and North Pacific who are confronting these very threshold issues of getting the capacity, bandwidth, into the country to begin with, and then we have these other issues around the Internet ecosystem, for example, Internet exchange points.
For countries like the Solomon Islands, Pacific island countries and even some countries in Africa where we are working, this problem of always having to go somewhere else for transit is a big issue. In the Solomons, for example, assuming that the submarine cable were in place, they would first have to buy capacity off of some provider, probably pipe, then they'd have to buy capacity off of Southern Cross.
So within that context, they're already expending a lot just to get basic capacity, basic bandwidth. I would be interested in knowing in that context the cost savings that a country like Solomon Islands could achieve, not only financial cost but in terms of what benefit, what further benefit the Internet that broadband capacity would have in terms of developing local content. Each of the Pacific island countries is quite unique.
Some of them have very polymorphous social cultural backgrounds and so local culture and language is incredibly important but also in terms of job growth, developing local industries, including local content industries.
A point that Bill didn't make when he was talking about the importance of broadcast or what we used to call broadcast, content development industry, it's actually one of the biggest industries in the world. It puts telecommunications and Internet provision to shame. Content development is big business in a lot of countries. It also is very important for social and economic cohesion in smaller countries. So I would be very interested to hear from some of you in the room who have dealt with these issues, I see Rohan and others, about your experience in both what works and doesn't work in developing IXPs because I'm actually very curious about this question of business models and governance. It's a relatively intellectually easy thing to do. Why isn't it happening more?
I think if we could draw lessons from experience, then we in our financing activities in countries around the world could better help identify the areas that could be addressed because it's not really a question of financing. No one will come to the World Bank for $50,000 loan to build out IXP infrastructure, yeah --
(Laughter)
-- we might include it as a sub-sub-subcomponent of a project but it's not a stand-alone issue so in that context I'd be very interested to hear what some of the real bottlenecks are so they can be addressed going forward.
Thank you.
>> BILL WOODCOCK: Thanks, David.
Next up is See Kiat Yeo, Director of Sales and Marketing for the Singapore Internet Exchange.
>> SEE KIAT YEO: Good morning.
My name is See Kiat Yeo and I run the national exchange that was just set up a couple years ago and I think Bill and Sam have touched on very much important the value of what an IXP can bring to an economy. What I want to try to do now is add a bit of flavor of what happened in Singapore and how we see going forward and challenges we face so that in the event any of you on the floor wanted to set up an IXP or additional IXPs, that is something to also think about.
So Singapore is a very small country. Very few eyeballs and limited lens space, so back in the 1990s when the Internet started off everybody was scrambling and like what Bill and Sam has mentioned everybody buys capacity to the U.S. to the extent the cost is overbearing and therefore some of the people in the community, vendors, universities, few of us got together and say, could we actually do something to keep some of the traffic local, especially, say, traffic between the local ISPs, locally so it does haven't to go through the very high cost of getting transit between the U.S. and bring it back to Singapore even to send an e-mail to my colleague.
So the universities and vendors came together and we set up what we had called a Singapore Open Exchange, SOX, SOX, still today in the university and the purpose was to keep traffic local, so that was back in the 1990s.
Then in 2000s when the dot com boom started, people started building data centers, you have big giants coming to Singapore and -- exchange point. Exchange points have its own benefit and it is a great way to actually support a typical data center or collocation business.
Then in the late 2000s Singapore embarks on a project to create a national broadband network where we get fiber to the home and offices. With creation of broadband to the home, broadband to the offices where you can run multiple like 1G, 10Gs, the industry started thinking hey, with all this capacity what do we do about so many content providers that have set up shop in Singapore, the likes of Microsofts, Googles, set up massive data centers to support the region. The industry came together with support of the government, the authority and says let's set up a Internet exchange point. Since it is for the benefit of the whole community we will set it up as a non-profit, non-profit.
And so, as the story goes, as we get more and more connectivities, cost of operation should just simply be maintained at a certain level and the cost of connecting to an exchange point for members should drop over time.
We only need that much more money to run exchange, whether 10 members, 50 members, 100, infrastructure is almost the same. So then the cost of participating actually drops and that actually bridges brings more value to the members. That is a big factor between a exchange and non-profit exchange. Today you see the European model of a membership-based approach is actually starting to take root exporting the idea to U.S. to see whether it's a model that makes sense for the economy.
From a non-profit basis, that is how we started. We started with zero members back in June of 2010, and the first member came on Board and he was alone for three months with no traffic. Since he's the only member and there's nobody to exchange traffic with. That is the toughest. Just like back in last month, we have finally hit about 40 members so it's growing slowly and surely but we still need the support of each and every one of the members to get exchange points going.
The other thing you notice is as I have mentioned earlier Singapore is very small, very few eyeballs, if you go to the SOX website on membership you'll find more than 3/4 actually comes from the region. So the likes of Indonesia, Thailand, Malaysia, Philippines, actually has connectivity to the points in the exchange. Maybe simply because they are operators themselves. They have submarine capacity so they can go in but I also make it a point when I talk to these members to tell them keep your local traffic local. Don't exchange Thai, keep it in Thailand, but if you want to take, say, if you want to take a piece of content which happens to have a major data center in Singapore by all means come and bring it back. That is what most of them do.
So we have actually online gaming companies in Singapore situated in Singapore developing it and so on and they make use of the exchange to support or distribute the games in the region. But I actually also tell them the same thing, eventually when your games get popular make sure you have a presence overseas as well so they can actually serve the local community better and faster and more efficiently. May not be the most cost effective but at least users in those countries can get access to content much more effectively.
So from where we are today, in Singapore typically you are looking at three exchanges. SOX is still in existence and serving education community so universities, polytechnics and schools, commercial one is -- and SGIX. So one of the challenges we face is what do we do in a place like Singapore where it's so small and you have three exchanges running around? One thing we did was early this year we actually had a connection up to SOX. So by connecting to SOX it helps the university network, they also don't have to keep investing in infrastructure to manage and maintain it because the equipment is already probably more than ten years old.
So we hope to reach out to the education community to help them, whether from the middle school, high school, colleges, so that the student population actually has access to it.
Then the other challenge we want to look at is now what do we do as an economic grouping in ASEAN where we have easily 10 regions in the region. On our own the countries may be pretty small compared to the U.S. or China or Japan but if the exchanges in the region could come together, we can be one step, first step keeping traffic local within each country. Next step can we keep traffic within ASEAN and after that, can we keep it in Asia and then how do we connect to other exchanges? All this is not about making money and making profits, all of it, but on building a community and more resilient Internet and that is where sometimes our views of the commercial offerings actually differs, right? We essentially want to be an exchange point -- besides delivering economic value to the economy, also actually to enhance on the resiliency of the network so that a single failure in a single location does not result in a catastrophe across a region.
So that's all I have. I am more than happy to have feedback later on. Thank you.
>> BILL WOODCOCK: Thanks. Next up is Garin Ganis who is one of the original participants in the Jakarta Internet Exchange which was I believe the first exchange in Asia to pass 100 participants, although many were very small networks.
>> GARIN GANIS: Oh, yes, okay. Thank you very much, Bill. Good morning, ladies and gentlemen. Just quickly because I was kind of summarizing our discussion I would agree, Bill was kind of thinking that with all these benefits mentioning why is it less than 50%, right, Bill, adoption of Internet exchanges worldwide? And perhaps about investments so David was kind of mentioning that, no, money may not be object after 50,000, maybe, right, because my experience in building I think in some countries like 10,000 is enough, you know, small switch, so when See Kiat was giving more benefits on this what's the problem with this Internet proliferation worldwide?
Maybe I take a different area, another component part of the exchange, building the human resource development of the Internet technology. I would tend to always split between like what I call layer one and seven issue, technical and that is layer eight and nine issue so seem to me economic is not the issue. I give two quick case studies in Indonesia. We started in 1997 out of Hobe, most back then came out from the Brittenberg systems, we had modems when Internet hits, 1995, 1996, naturally we are technical savvy would like to go to that area so we are early adopters.
We actually put on with these protocols tried to do exchange and at that time in Jakarta, only about seven or eight, so the government has not really stepped in there so the idea was like technical playing around with it. So has to be interest.
Not an economic value first, service is technical. That's our experience.
By 1998, of course we are competing with incumbents, build telcos, they are not doing this voluntary Internet action stuff. That was 1998 and quickly by 2005, about 100 ISPs already joining us. That's just an example of maybe we should start not talking about business for the benefit but just playing with technical areas first. I repeat the idea in 2000. About three years ago I was invited by ITU to do implementation design for Afghanistan Cable Internet Exchange. Same issues. I was there three weeks and of course the government would kind of help but ultimately I gather all ISPs and guess what? This is not even technical. One ISP said to me, Garin, make sure the Internet action filters everything, pornography, fine, sir.
In the afternoon I went to another ISP and they said Garin, make sure no filtering, we want all these things because we have military, they need entertainment and --
(Laughter)
-- provide these things. I said okay. So I have to design the other and the other, and then by the end of the day I realize wow, they don't even know what Internet action is. So I do some workshop, finally I said no, sir, filtering is on your protocol not even -- just basically a switch. Then I realize doing some interface at that time there is only one -- not -- one Cisco academy and only for CCNA, the lowest certification. So here we go we have issues. Country doesn't even have proper network engineers. I've been pushing this idea all along that you have to have some technical staff. It's not certification-based but basic technical people. Going through the ranks of technical issues you have to understand of course routing configuration but along the way TCIP and again this was amazing part of the whole skill set in that country.
Then finally -- not easy to do, that's why there are teams here. Basically I look at two different countries, all underdeveloped countries. So IT was willing to give anything money, Bill and PCH would like to give money. At the end I said 5,000, it's enough for starter, layer two, switching design, very simple but it requires technical on those participants to do. We ended up inviting them to Indonesia to see three operations of Internet, different flavors at the time. So finally they came to Indonesia for about a week and then went back and set up the Internet exchange.
I think again I am just contrasting not just economic issue, perhaps the best -- better way to do this is to push this, by the way, just wave to me, anybody who doesn't have any Internet in the countries, respective countries right now in this room. Anybody? That's one. From?
>> (Speaking off-mic)
>> GARIN GANIS: That's a classic Indonesian. Any more Internet Exchange -- (?) doesn't have not in this, so Bill, you have to do a better job next time to invite the other 50%, you know. But I guess I have to stop here because then it's clearly it's about economics. Go ahead, thank you.
>> BILL WOODCOCK: So I think at this point we'll switch over to discussion and I'll kick it off by addressing a couple of the points that I heard brought up by our speakers. Also I'd like to let people know in the audience we have several people who have funded major Internet exchanges. Nikhil Yard in the checked shirt there, one of the founders of the -- Internext Exchange. (Inaudible) and building the Tunisian Internet Exchange points, Elena is running the Russian Exchange points which are very, very large by size and the guy in the red shirt who has been popping in and out is Pinder Wong who built the Hong Kong which was the first in Asia, also the first academically hosted, first exchange point hosted by a university.
So a couple points I heard both David and Garin touched on the cost of setting up exchange points and mentioned numbers between 5,000 and $50,000 U.S., PCH's studies indicate that 90% of exchange points are built for between $8,000 and $40,000 U.S. so five to 50 would cover 95% of exchanges. So there have been people in the ITU who have said $10 million to set one up, this is ridiculous, right. You can get a very good exchange point up for well under $10,000. Second point was addressing something that David said about cost savings and as David and Sam are both economists of sorts, I think this is probably a shorthand and it's a shorthand I use as well but in, if we were really talking about cost savings what that would mean would be economic contraction, not actually what happens and not actually what's interesting.
Right. So this is a shorthand and it's not a literal truth. What we're looking for is not cost savings; we don't want people to spend less on the Internet. What we want is for the ratio of what they are spending to what they are getting out of it. The value they get out to improve. By analogy think about cars. Say you are in a developing country where it's really, really expensive to get a car in. Getting a car costs $100,000 for even a not very good car. So very few people can afford cars. Cars are expensive, not very many cars get sold. We have a medium spend on cars because we have a small number of sales times very high price.
That's the way the Internet is in a lot of developing countries. Very few people using a very small amount of it because it's really expensive. We are not trying to reduce the amount of money spent on the Internet because that would be great for those few people. What we are trying to do is bring the price of Internet access down so more people can afford to spend the money to buy it which means that what we have is a whole lot of people who can suddenly afford to buy a medium-priced car so the amount of money getting spent is suddenly huge compared to before but the social benefit of everybody getting to have a car if they want one is high. Maybe social benefit of everybody having a car is not such a good thing but it's an analogy. Everyone getting access to the information super highway what it was called back when I started is a really big social benefit. Having the price low enough that many people can afford it is really what we're talking about here.
So with that, I'd like to see if we have questions or points of discussion or whatever.
Yes, sir.
>> Yes, I'm Bernard Sedaka from Lebanon ISOC chapter. I am really, really interested in the Internet exchange points and really believe in it and we already have one of the first in the Middle East set up in Beirut but my problem is this: For a exchange point to be effective we need local content. So without local content the exchange point is like there just so we make sure it's there.
But from the point of view of developing country for the Internet exchange point to be effective from my point of view is we need to start to have better bandwidth at the beginning and then later on once we have more local content, then we shift from outside to the inside where we actually get the benefit of the Internet exchange point.
In Lebanon, the government controls the international gateway, so all ISPs have to pass by the government in order to get content from anywhere in the world. There is no local content so the exchange point is there but actually not doing anything.
Specifically when the government ISP is not also connected to the IXP so if I am to send an e-mail to someone using connection from the government, the governmental ISP I have to go to the Internet and come back so the government is charging me twice what I'm supposed to be sending locally even though there is an IXP and government is not connected to it. So this is a big dilemma.
Second one is for a good IXP to be efficient we need good Internet bandwidth. If I want to install in Beirut let's say a Google cache, simple Google cache as the Palestinians did as well, this will be very effective and very helpful for the country so I don't have to go to the Internet in order to get the cache. Instead of clicking a billion click or like 10,000 clicks per day for one single link I click it and get it from the Google cache.
I cannot do it because the local bandwidth, international bandwidth is not good enough. How can we solve this dilemma? Thank you.
>> BILL WOODCOCK: Since I have a slide I use a lot that touches pretty directly on what you're talking about I pulled it up and I'll give a quick crack at that and then turn it over to the other panelists.
So the exchange point is really one portion of an ecosystem. It's the cheapest and easiest and fastest part to get right. It's often the first part that gets done but the Internet has doubled in size every ten and a half months for 30 years and in order to continue that rate of growth you have to keep touching each portion of this ecosystem every couple years and make a tenfold upgrade in the performance of it and capacity of it.
So when we got the (inaudible) up and running, members voted to put it up on a hilltop where they could all reach it by microwave but the problem is microwave tops out in practical terms maybe 155 megabits which is -- seemed good compared to two megabit circuits at the time but is not that great compared to the 10 gigs you can light up on your own fiber and because it's so distant up there, it's difficult to get fiber up to it.
I don't know whether -- do you know whether that's happened, whether anybody has hauled fiber up there?
>> Well, I have no specific data about that. I'll ask them.
>> Last I knew, if you remember the local data lease lines had to be from -- who had a monopoly on leasing connections, using microwaves so part of this is not technical, it's regulatory. In addition to the monopoly with international bandwidth.
>> BILL WOODCOCK: You have to touch on each of these things and fix each of these problems because each one can be a bottleneck. You have a perfectly functional exchange with high -- with microwave links very limited capacity to get to it and the reason it was put there was because that was better than the alternative at the time which was the two mg circuits which were very expensive so that was a local loop problem and regulatory problem. In addition, you have the sort of critical infrastructure problem of these things people really need to get to which some of them are at the IX and some are not. You have the data center issue, IX is up at a building that is used for a lot of business park kind of things but is not really a data center so there's nowhere to go and put a lot of servers if you want to. And it's been what, like six or seven years now, right, so during that time you should have made it all the way around this loop twice, right?
You should have had two tenfold upgrades in each portion of this loop during that time. I suspect basically what's happened here is the local loop problem has not gotten solved and the regulatory hasn't gotten solved so the other things are relatively minor by comparison because those two are the next bottlenecks facing you.
The Beirut IX, yeah, it's an example of what can happen with IXs which is you do everything right in that moment and then things kind of plateau, like it solves an immediate problem, things get a little better but then the improvement stops because people don't go on and fix all the other problems.
IXP is no longer the bottleneck. There are other things that are the bottleneck so the IX can't solve regulatory problems. You have to talk with your regulator to solve that.
Other people want to take a crack at answering that?
>> Just one quick point, you made the point maybe you don't have enough local content. You need that to create an IXP, that's what you kind of said. You have lots of content.
>> Lots but hosted outside.
>> That's the same issue we have with Mexico. Mexico creates an enormous amount of content but all hosted; Mexico creates an enormous amount but all hosted across the border. No exchange point. Government is starting to stress that, the president said we need an Internet exchange point but it is incredulous to me there is no exchange point in Mexico, incredible. They host the content across the border, because it's cheaper because the like all ISPs with monopoly power, basically won't play ball and that's the regulatory problem.
>> Without commenting on the specifics of the problem raised by the question, I would just underscore the point that was made by the questioner and by Bill that there are probably exogenous factors that need to be taken into account in why in that certain case an IXP is there but not functioning the way it should be. I would submit that in that case it's probably not even a regulatory problem but a political problem.
>> SEE KIAT YEO: Are any of you operators in this room? Okay. You guys happen to be in your own countries? Okay. No, okay. Let me just add on to what they said about Beirut. Singapore. Highly connected and all that. We always have problems with incumbents. I have actually -- incumbents are connected but they refuse to peer locally. That's the problem we have. So the point about regulatory certainly plays a part and why does incumbent refuse to peer at exchange point, smaller IXPs for example? Simply because of the fear of loss of revenue, fear of loss of market share. They would rather say charge content providers a fee to connect to the local eyeballs rather than for the benefit of the users.
So when you get a point up and running it's always very important to figure out how to work with the incumbent, get them on board and to buy in so as of today in Singapore it is --
and you don't see them, they don't -- they don't pay the members I have, they would rather sell them IP transit for revenue. And I was thinking that maybe in the next five years, next ten years things will change and move and they will see it differently. So that's my two cents worth.
>> Sanjay from APNIC. That has been my observation as well. People of different sizes don't want to talk to one another. If they are on the same size, they want to exchange traffic because they are as strong, you know, and so the top level, usually they do a bilateral, but if you can actually create a different tier of Internet exchange so you can go down and say medium level player, you play on this ground. Lower-level, you can play on this ground. And once these different levels are connected and create enough traffic, then these three will realize that, oh, we almost have the same traffic. It's just there's only a few players on the top, slightly more middle and a lot of people at the lower and then once the three layers say, oh, we have the same level, then they connect. Almost like you guys and SOX, well, maybe we should interconnect.
I think part of addressing the problem is realizing in the market, this is market reality, you cannot pass through. It's market reality that big people only want to talk to big people. Small people talk to small people. Use that to the advantage and create different tier of IXPs. Thanks.
>> On that, the sizes of providers do self-segregate in exactly the way you are saying. You don't need different Internet exchange points to support that. They can do that within the context of a single exchange point and generally do, yeah, exactly with the layer two exchange point. What we have seen that is surprising actually given the popular wisdom for a long time is that route servers, layer three route servers are actually becoming very prevalent in the number of sessions that exist. So a route server is a thing that allows many ISPs to tier with each other with a minimum configuration but also minimum amount of control. You can't have different policy for different people you're peering with if you are using a route server.
So the popular wisdom for a long time was big carriers would say, oh, only little insignificant ISPs use route servers, they are something us big companies would never touch; our lawyers would never allow us to muddy our feet in such thing. Right.
But in fact if you look at big route servers in Frankfurt and a hue huge one in Warsaw and big one in Hong Kong and big one in the Czech Republic, these have as many as 100 or 150 participants all peering with each other. That forms a full mesh of peering sessions, huge number. So that is exactly what Sanjay is saying. A whole lot of little guys all aggregate together and suddenly the value of that aggregate becomes very high so big guy looks and says well I won't admit it to the other big guys but if I connect to that route server suddenly I'll have a lot more traffic to sell to my customers. So route servers actually worked out very, very well in that way.
Pinder has something to say.
>> PINDER WONG: Pinder Wong from Hong Kong. I mean, the route server thing is a great analogy because what you thought happened and what actually happened, again, things change as your diagram illustrates. Just wondering, I'm sorry to be in and out but you may have mentioned it. Do you have data on sort of you mentioned layer two. Layer three. How does that pan out in practice in terms of exchanges? How many? Do you have data on that?
>> BILL WOODCOCK: PCH operates a directory of exchange points so we have the numbers on that. There are essentially no remaining operating exchanges that are layer three only. There are none that are mandatory layer three. There are a lot of essentially all exchange points have a layer two switch fabric but many now have layer three route servers alongside.
And Nick, is that something you want to maybe talk a little bit about, how route servers work? Nick is one of the world's experts in that.
>> Hi, Nick Hilliard from the Technical Community. Yeah, route servers offer a way for participants to exchange routing information between each other. There is a certain level of control there. Not a huge level of control. So it tends to be more useful for smaller participants. Larger participants of Internet exchange will generally tend to think well they prefer to use bilateral interconnection because it suits their purposes a lot better.
There are scaling issues associated with route servers on really big exchanges. You can run into problems with them. But they're problems you can deal with if they are managed correctly. Certainly at INEX we found them to be incredibly important from a strategic point of view. We actually monitor our interconnection rates, the number of bilateral about multilateral peering sessions and what we've noticed is the vast majority of the interconnection of the exchange is actually multilateral which means it goes over the route servers.
So this is a huge boon to the value of an Internet exchange. Any one out there not using route servers really needs to consider whether they ought to use them in the future because they add so much value.
>> I would like to comment on the cost thing because I heard very good comments about it so I think there is a lot of misunderstanding I would like to -- this is one of the most thing I think it's very -- you heard the message from my friend from Lebanon but it's the same thing for Tunis. ITI played for years a role of exchange without really knowing the value, without being in -- according to best practices. They didn't consider best practices for long years.
And the thing mentioned by Garin about filtering and not filtering, this is our concern for years also in ITI. The thing we are today building a good exchange point in terms of best practices, it is for sure we know that it's not a matter of cost. It's very easy. We don't have problems with paying all those switches, we are familiar with them. We pay a lot, we have a lot of good infrastructure also in layer three as mentioned but we want to have neutral Internet exchange point. That's where the matter of ownership and who is spending money for these points is very, very important.
Second thing, that it is very important and was concerned for that for long time. It is about the data centers. In our countries the problem is where to host the Internet exchange. Okay we want one. We already hosted in data centers but is it accessible to the other players. Is it open? This is very good question. In Tunisia we had -- to have data centers everywhere. Now we have one so we started by building one because it's neutral, promoted it, Bill helped a lot for this and now we're thinking about Tunis but the problem with Tunis it's not -- only operators incumbent operators have data centers.
Second would be about cost. If you could talk about cost, it would be how to build a neutral Internet exchange point. In Sweden they have an experience that governments supported that. Is it right approach to make it in developing countries?
I know Sweden is very well organised and today very good switch exchange point there. But for developing countries how we can deal with this to promote neutral in order to have very good exchange point like in the one -- solve the problems of Lebanon, for example.
>> A quick comment. As I mentioned, the success of -- pushed by communities so do you have any Internet community Internet service providers? Because their function is supposed to be a lobbyist, active lobbyist and government should not be -- they should focus on being regulators, seems to me in Lebanon they mingle with business as well but I don't know. But in Indonesia, not just -- but governments kind of confused sometimes. This is where -- very strong in pushing so technically and being -- that's the first venue we perhaps you should follow actively. Quick comment on that, yeah.
>> BILL WOODCOCK: Really quickly on Sweden, what Sweden did is --
(Lost audio)
-- market rate but then the government used tax money to take the exchange points and locate them inside nuclear hardened bomb shelters and put a lot of redundant fiber in and out.
I think Garin had something he wanted to say.
>> Thank you. I found that obviously Bill's presentation actually raises challenging points because one could be you are going to a mercantilist framework that suggests that the more you sell and the less you buy, the world will be a better place, which I find quite somewhat disturbing.
And also I think when you talk about privacy issues, government control issues, well there are some people in this room who trust their governments and there are some people who sort of highest distrust is leveled against their government as against other people's government. For example, in my organisation, which basically is a regional organisation which works with as a virtual organisation, my rule has been keep our data anywhere but in this country. Keep it anywhere, I don't care, preferably two or three places but don't keep it here because this is the only place anybody can arrest me, right.
So you know, I think you need to factor these nuances into the conversation. Back in 1998 I was a telecom regulator and ISPs came to me and said you know we are having all kinds of difficulties, I suggested what about an IXP -- Internet exchange point, remember the days are long past, right, this is when your ISP used to give you an e-mail address, an e-mail was a big deal. Right. Not like now. Where everybody's e-mail is sitting on some cloud server somewhere. Right. Of course we had the classic problem that Singapore -- the speaker from Singapore explained which is rather obstinate and uncooperative incumbent. But you see, as a regulator --
(Audio lost)
-- there is an Internet exchange in Sri Lanka but in reality it's like Lebanon, most goes outside because there are too many transaction costs involved. And also because much of the I think we have to look at this. In terms of where is the content that people are using? You look at the Alexa or wherever and see, people are looking at Google, Yahoo!, looking at YouTube et cetera et cetera et cetera. Where is local content? The top 10 or top 20 or top 50? If that is the case how much sense does it make to have this one-size-fits-all solution. And what operators are looking at is a sort of composite issue which has peering costs, IT transit costs, whole series of things that affect their bottom line because what I fear is if this argument goes too far you may have governments, particularly with the new national Internet lodge gaining ground in the last few months mandating that ISPs -- you might become very popular, that all governments will mandate we must have national IXPs, preferably controlled by the government, right?
So I hope we don't go that route. People have been given options. People should have the option of doing things within the country, they should have the option of doing things outside the country. Because otherwise what I fear is that some of the needed action for the international -- bringing down the cost, that we are working on -- just to give an example, IP transit in Singapore versus Philippines, it's a factor of ten. More costly in Philippines. But then you take IP transit in Singapore versus London, Singapore is still very expensive compared to London. These things need to be addressed while we create the conditions for people to do their own IXPs, and I hope whatever we have tried as telecom regulators we have not really done a good job with ordinary voice interconnection in circuit switched networks throughout the world.
I hope, I mean, compared to that, Internet interconnection is working pretty well. Be a little careful that you don't push it too hard that it goes and starts assuming some of the problems that we have experienced on the voice side.
>> BILL WOODCOCK: So Garin made quite a few points and brought several things up. Any of you guys want to touch on any of those?
>> Just comment on the telecom regulatory question you raised. Because about the same time as the ISPs were approaching you in Sri Lanka in my home country Australia some of the big ISPs were going to the regulator and saying we have one dominant player here who will not peer with us. You should make them because this is just like telecom's interconnection and you make them interconnection with us. Regulator said, whoa, yeah, okay. They chose the next biggest three players so the four largest players were then peering with each other based on a decision by the telecom regulator.
Now, you could look at that in terms of what we knew about telecoms at the time and say we knew this made sense in Telecom World, probably makes sense in the Internet world. What happened was as a result of that, the next largest players were certainly cut out of the market and it took them a huge amount of time to build competing infrastructure that would have developed if the next three biggest players had basically not had this entre to the biggest player and developed infrastructure and learned the lessons and developed competing infrastructure quicker so it probably set back the Australian market by a decade until other players started developing competitive infrastructure. That's one of the tradeoffs.
>> BILL WOODCOCK: I think two things that you brought up I can touch on briefly. One is the question of whether stagnating IXPs mean it was not the right thing there, was not an appropriate solution.
I think the exchange point is a necessary precondition, building block, and there are a bunch of other necessary preconditions to having a good market. A bunch of other building blocks before you get a complete structure. So I think saying well, this is not working perfectly, not solving all problems, this is not what's interesting there. I think saying we got this part done, now we need to turn and look at the other parts and do them equally well is probably what the appropriate lesson to drive there is.
The other issue of privacy and saying, well, the government of your own country is probably the one you are most worried about snooping on your traffic, absolutely that's true. The question is not is it the government of your own country or the government of another country; it's the question whether it's the government of your own country and the government of some unknown set of other countries.
So this is exactly the same problem as we have right now with the certificate authority system failure versus DANE. So DANE doesn't say nobody can break your certificate. What it says is only your TLD provider and the route can break it. So instead of having a set of 500 random companies around the world, any one of which can issue an invalid cert, now there's a specific known set and it may be that your ccTLD operator is the one you are most worried about but at least you know who to look at.
Same thing with spying and exchange points, right. That there are very few large exchange points that don't have wiretap access from the national government in which they're operated. But the question is more do you want your traffic going through that one where you know who is seeing it or do you want it going through some random set where you have no clue who all is seeing it and how far they are sharing it?
>> Something else (inaudible) raised about the content being offshore. I imagine that when you create an IXP, because it's such an essential part of the whole ecosystem, by doing that eventually you do attract the Googles or Akamais to locate facilities once you get to a certain level. And so that probably is an important point.
>> BILL WOODCOCK: I guess we're about at the time now. I think if anybody has one more question we can tackle it; otherwise, we could call an end to the session.
Yes, sir, in the back.
>> One comment on that, on the last comment. In New Zealand we found both Akamai and Google in early stages said to some of the ISPs if they wanted to install a network, one of the preconditions of getting that in was to attach it to the exchange because that would drive up the required scale so it does happen.
>> BILL WOODCOCK: Pinder.
>> PINDER WONG: Sorry, Hong Kong Internext Exchange was in the news recently. Question here is in terms of equipment ordered. How do we know -- I guess are there any exchange points that practice some degree of security for the equipment that gets attached?
>> BILL WOODCOCK: I think most small exchanges, all participants wind up seeing all of the equipment because it's all right there in one cabinet. The large exchanges are the ones that tend to be wire-tapped. Then you have the sort of weird outliers like the Cambodian Exchange which operated completely illegally for 10 years where because it was against the law and there were governments investigators looking for it to try to shut it down, yes, they did very active auditing of what was connected to it. And with the Beirut Exchange that was an interesting case because one of the big political issues in starting that exchange was should only government-licensed IXPs be allowed to connect or should both government-licensed and Hezbollah-backed ISPs be allowed to connect?
Originally it started out as only the government ones because they had sort of the majority in the conversation at the beginning and then I think it opened up over time as people realized that it didn't make sense to have domestic Lebanese traffic going over satellite into Europe and back again over a cable and blah, blah, blah. I think the really big exchanges are the ones that have government wiretap on them and that's kind of taken for granted. Little ones everybody sees and they tend not to be tapped.
So all right. Well, I thank everyone very much for attending and participating and the good conversation. I think speaking for all of the panelists here, if you have further questions and want to talk with any of us, we're all very open to that. Thank you very much for your time.
(Applause)
(Session concluded)
* * *
This text is being provided in a rough draft format. Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
* * *